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The  primary  objective  of  the  Simultaneous  Sensor  Presentation 
Techniques  Study  was  to  investigate  the  utility  and  feasibility  of  combining 
imagery  from  multiple  airborne  sensors  and  data  sources  on  a single  real- 
time display.  To  provide  a framework  for  the  study,  a single-seat,  high 
performance  attack  aircraft  was  assumed,  having  a sophisticated  avionics 
suite,  ordnance  capability  for  the  complete  range  of  conventional,  laser/ 
KO-guided  and  anti- radiation  weapons,  and  a full  complement  of  advanced 
sensors  for  the  air-to-ground  (and  air-to-air)  combat  role. 

To  identify  the  types  of  information  which  might  be  synergistically 
combined  and  displayed  during  a typical  mission,  the  interdiction  mission  was 
selected  for  detailed  analysis.  Interdiction  depends  heavily  on  precise  navi 
gation,  penetration  tactics,  and  self-contained  target  acquisition  and  weapon 
delivery.  This  places  a severe  task  loading  on  the  pilot.  Because  his  per- 
formance of  visual  and  manual  tasks  during  these  critical  mission  phases 
requires  the  simultaneous  utilization  of  a variety  of  real-time  sensors,  a 
priori  data,  and  possibly  data-linked  information,  it  is  here  that  the  opti- 
mized display  of  multiple  sets  of  information  may  best  serve  the  operator. 

By  consideration  of  the  mission  profiles,  the  operator  tasks  required 
per  mission  phase,  and  the  capabilities  of  the  baseline  sensors  available  to 
the  operator,  an  extensive  number  of  possible  data  combinations  emerged. 

A qualitative  evaluation  of  these  combinations,  based  on  estimated  utility  to 
the  operator  and  complexity  of  implementation,  has  resulted  in  several  can- 
didate simultaneous  data  presentations  which  are  recommended  for  further 
study: 

• The  simultaneous  display  of  real-time  radar  ground  map  and 
stored  cartography  and  waypoint  data  to  assist  in  accurate 
navigation  to  the  prebriefed  target  area. 

• The  simultaneous  display  of  real-time  radar  ground  map  al\d 
stored  reconnaissance  imagery  of  prebriefed  targets  for  pre- 
liminary target  acquisition  at  long  range,  for  target  verification, 
and  for  offset  aimpoint  designation. 

• The  simultaneous  display  of  real-time  radar  ground  map  and 
real-time  FI.IR  data  at  long  stand-off  ranges  to  provide  thermal 
gradient  indications  for  cuing  the  operator  to  possible  target^ 
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on  tilt-  radar  display.  This  could  be  particularly  valuable  for 
the  acquisition  of  large  target  complexes  or  for  hot  spot 
detections  corresponding  to  targets  of  opportunity  such  as 
moving  vehicles. 

• The  simultaneous  display  of  real-time  F LIR  imagery  and  radar 
glints  for  cuing  the  operator  to  man-made  objects  on  the  FUR 
display  which  could  be  cold  or  hot. 

• The  simultaneous  display  of  real-time  F LIR  and  real-time  TV, 
processed  and  mixed  in  an  optimum  fashion  such  that  certain 
types  of  targets  may  be  acquired  in  shorter  time  or  at  longer 
range. 

A general  system  implementation  for  achieving  all  of  these  combina- 
tions in  real  time  has  been  conceived.  The  system  will  accept  inputs  from 
any  of  the  above  sensors  and  data  sources,  will  spatially  warp  and  register 
any  two  desired  images,  will  perform  various  selectable  enhancing  opera- 
tions on  one  or  both  images,  and  will  present  the  combined  data  on  a single 
display.  The  relative  complexity  of  mechanizing  each  individual  combination 
is  also  considered.  In  practice,  an  operational  system  will  undoubtedly 
implement  only  a limited  number  of  the  processing  operations  shown  in  the 
general  system,  depending  on  the  types  of  information  to  be  combined,  spe- 
cific performance  and  accuracy  requirements,  desired  flexibility,  and  per- 
missible hardware  size,  weight,  and  power.  It  should  be  noted  that  much 
of  the  hardware  shown  in  the  general  system  implementation  (and  thus  the 
austere  systems  also)  may  already  be  onboard  a high  performance  tactical 
aircraft  and  may  be  time-shared  for  use  in  the  display  of  multisensor  infor- 
mation. Thus,  the  inco rpo ration  of  a simultaneous  presentation  capability 
need  not  mean  a major  addition  of  avionics.  Also,  all  of  the  elements  identi- 
fied in  the  general  and  austere  systems  represent  equipment  that  is  available 
or  presently  in  development. 

Future  efforts  to  confirm  and  quantify  the  results  of  this  study  should 
include:  1)  the  acquisition  of  simultaneous  coincident  coverage  multisensor 

imagery  of  tactical  targets  in  various  backgrounds;  2)  the  spatial  warping  and 
registration  of  that  imagery;  3)  the  application  of  a variety  of  image  process- 
ing and  enhancing  operations;  4)  the  display  o the  registered,  processed  data 
with  a variety  of  video  mixing  techniques  and  false  color  combinations,  and 
5)  the  quantitative  evaluation  of  the  resulting  simultaneously  presented 
imaigery  by  a human  factors  study. 
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1.0  INTRODUCTION 


1.  1 BACKGROUND 

Future  military  engagements  in  all  probability  will  see  the  U.  S.  forces 
handicapped  by  1)  the  enemy  enjoying  global  and  local  numerical  superiority 
and  Z)  the  enemy  having  the  advantage  of  first  strike  and,  consequently, 
momentum.  Thus,  the  U.  S.  must  depend  heavily  on  using  superior  technology 
and  sophistication  in  its  tactical  equipment  to  generate  highly  mobile  and  con- 
centrated firepower,  so  that  the  enemy  thrust  can  be  first  blunted,  then 
repelled. 

The  superior  technology  and  sophistication  in  equipment  must  manifest 
itself  in  all  combat  phases,  from  reconnaissance  and  staging  to  navigation, 
target  acquisition,  weapon  delivery  and  damage  assessment.  The  sensor 
equipment  involved  is  usually  related  to  extending  the  man's  capability  to 
sense  the  battlefield  situation,  typically  by  means  of  passive  and  active  sen- 
sors and  associated  data  processors  and  displays.  In  most  cases,  a man  will 
remain  in  the  loop  to  monitor  and  interpret  the  output  of  these  sensors. 

Experience  has  shown  that  increasing  the  number  of  airborne  and 
groundbased  sensors  can  result  in  an  increased  probability  of  successful 
target  acquisition  because  of  a target's  multi  - spectral  attributes.  The  multi- 
sensor approach  circumvents  enemy  attempts  to  make  use  of  stealth  and  cam- 
ouflage, since  all  target  signatures  cannot  be  viably  reduced  simultaneously. 
The  advantages  of  multisensor  utilization  in  air-to-ground  combat  were  well 
demonstrated  in  the  highly  successful  AC-  130  Gunship  operation  in  Vietnam, 
followed  by  Tropic  Moon  III,  and  leading  to  the  current  Target  Recognition/ 
Attack  Multisensor  (TRAM,  A-6E  and  A-7E)  in  the  Navy,  and  Pave  Tack  and 
Quick  Strike  Reconnaissance  (QSR)  in  the  Air  Force. 

As  the  number  of  sensors  and  avionic  subsystems  increases,  however, 
the  amount  of  information  which  must  be  displayed,  controlled,  anc.  assim- 
ilated by  human  observers  is  also  increasing.  Thus,  a severe  task  loading 
problem  is  developing,  particularly  in  attack  aircraft  where  mission  success 
is  strongly  dependent  on  penetration  tactics  and  self-contained  target  acquisi- 
tion and  weapon  delivery.  The  problem  is  intensified  during  nighttime  mis- 
sions. In  addition,  many  of  the  weapon  platforms  are  single  seat  aircraft 
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(A-  7,  A- 10,  F - IS,  F-16,  F- 18/A- 18.  A-4M,  etc.),  the reby  plat ing  the 
burden  of  navigation,  threat  detection  and  defensive  action,  target  acquisition 
and  weapon  delivery  on  a single  operator. 

There  is  an  immediate  need  to  maintain  the  advantages  of  multisensor 
target  acquisition  while  coping  with  the  task  loading  limitations  imposed  by 
single  seat  air-to-ground  combat.  In  the  past,  each  sensor  has  typic  ally  had 
its  own  dedicated  display  and  observer  (Gunship,  for  example).  Human  fac- 
tors studies  have  shown,  however,  that  the  grouping  of  several  sensor  dis- 
plays for  target  recognition  purposes  by  a single  observer  utilizing  several 
sensors  actually  results  in  increased  recognition  time,  probably  because  the 
observer  spends  much  of  his  time  searching  the  displays.  These  results, 
combined  with  instrument  panel  space  limitations,  have  led  to  time- sharing 
of  sensor  inputs  on  a single  common  display  surface.  This  technique  requires 
a visual  "reorientation  at  each  sensor  change,  and  also  the  manual  switching 
of  sensor  images  for  best"  sensor  selection.  The  system  could  be  auto- 
mated; however,  the  programming  of  such  timesharing  may  be  quite  critical. 
Clearly,  innovative-  techniques  arc-  required  to  provide  the  pilot  with  the 
salient  information  he  needs  from  each  sensor  at  appropriate  points  in  the 
mission  timeline  and  in  a readily  interpretable  manner. 

The  study  reported  herein  examined  techniques  for  combining  multi- 
ple sensor  data  in  an  effective  and  timely  manner  for  simultaneous  display  on 
a single  display  surface.  A key  objective  was  to  define  those  simultaneous 
sensor  presentations  (using  techniques  which  can  be  practicably  implemented) 
which  have  the  potential  to  increase  the  probability  of  successful  target 
acquisition  and  weapon  delivery,  improve  survivability,  and  achieve  a high 
level  of  mission  performance  with  a reasonable  operator  task  loading. 

1.2  STUDY  APPROACH 

This  general  problem  has  been  rendered  addressable  in  this  study  by 
concentrating  on  1)  a high  performance,  single-seat  attack  aircraft,  and 
2)  a single  class  of  airborne  missions  which  imposes  a severe  task  loading 
on  the  operator,  i.  e.  , the  interdiction  mission  for  prebriefed  fixed  targets 
and  for  targets  of  opportunity  in  prebriefed  target- rich  areas.  The  study 
approach  was  to  divide  the  interdiction  mission  into  key  phases,  define  the 
operator  tasks  per  mission  phase,  and  determine  the  information  required 
to  perform  those  tasks. 
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An  extensive  list  of  applicable  sensors  was  surveyed  in  terms  of 
operational  parameters,  detectability  of  types  of  targets,  and  typical  range 
capabilities.  The  information  requirements  of  the  various  interdiction  mis- 
sion phases  were  then  used  to  select  or  "filter"  the  list  of  candidate  sensors 
and  sensor  modes.  This  approach  placed  the  performance  requirements  and 
sensor  selection  in  context,  and  reduced  the  large  set  of  applicable  sensors 
to  a small  set  of  baseline  sensors  which  are  reasonably  affordable  and  hence 
realistic. 

The  baseline  set  of  sensors  were  then  analyzed  with  respect  to  possible 
simultaneous  presentation  for  each  phase  of  the  mission  including  cruise/ 
navigation,  preliminary  target  acquisition,  run-in  to  target,  and  target 
acquisition/weapon  delivery.  From  this  analysis  emerged  a primary  or  "lead 
sensor  for  each  mission  phase,  a set  of  secondary  sensors,  and  a set  of 
required  or  desired  supplementary  data.  Various  techniques  were  con- 
sidered for  combining  primary  and  secondary  sensor  data  on  a single  display 
including  1)  overlaying  one  sensor  image  upon  another,  2)  deriving  "highlight" 
data  from  one  sensor  and  displaying  it  as  an  overlay  on  another,  and  3)  using 
overlay  symbology  to  represent  target  or  threat  locations  as  determined  from 
onboard  sensors,  as  data-linked  to  the  aircraft,  or  as  called  up  from  a priori 
data  stored  in  the  aircraft  data  base.  (A  literature  search  was  conducted  to 
ensure  that  results  of  previous  studies  in  this  area  could  be  evaluated  and 
taken  into  consideration  in  the  present  study  where  applicable.  ) 

The  relative  merits  of  these  candidate  combinations  were  subjectively 
evaluated  by  means  of  synthesized  and  simulated  video  combinations,  stat- 
ically displayed  in  black  and  white  and  in  color.  Various  processing  or 
enhancing  operations  which  can  be  applied  within  each  combination  were  also 
considered.  Based  on  an  examination  of  each  possible  combination  in  terms 
of  potential  utility  to  the  pilot  and  relative  complexity  of  implementation,  the 
most  promising  combinations  that  emerged  include  the  simultaneous  presenta- 
tion of  1)  radar  video  and  cartography,  2)  radar  video  and  stored  reconnais- 
sance imagery,  3)  radar  video  and  F LIR  or  TV  video,  and  4)  FL1R  video  and 
TV  video.  In  all  cases,  supplementary  data  can  be  displayed  by  the  overlay 
of  appropriate  symbology. 

The  implementation  of  the  simultaneous  sensor  presentations  was 
examined  in  terms  of:  preprocessing  of  the  two  sensor  video  to  make  them 
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format-compatible;  registration  of  the  multisensor  images  or  data;  image 
processing  to  suitably  cnham  c or  extract  highlight  information  from  either 
sensor  image;  and  finally  the  mixing  and  display  of  the  two  registered,  pro- 
cessed sensor  video  in  an  optimum  manner.  A multisensor  system  concept 
was  described  for  accomplishing  the  various  correction  anti  processing  opera- 
tions in  real  time  with  flyable  equipment.  The  system  is  modular  in  nature, 
microprocessor  controlled  and  highly  programmable,  ami  makes  maximum 
use  of  processing/display  elements  already  onboard  advanced  aircraft  or 
scheduled  for  future  incorporation.  In  addition  to  the-  general  system  imple- 
mentation, the  relative  complexity  of  implementing  individual  candidate  com- 
binations was  also  examined.  The  study  concludes  with  recommendations  for 
future  effort,  including  hardware  development,  the  simultaneous  acquisition 
of  coincident  coverage  imagery  by  multiple  sensors,  and  a subsequent  dynamic 
simulation  and  evaluation  of  the  recommended  multisensor  combinations. 

The  following  sections  present  the  mission  analysis,  the  methodology 
used  to  select  multisensor  candidates,  the  description  of  candidate  simulta- 
neous sensor  presentations  as  a function  of  mission  phase  (including  synthe- 
sized imagery),  the  image  processing  considerations  for  multisensor  combin- 
ations, implementation  conside rations  and  typical  system  operation,  and 
summary  and  conclusions.  The  results  of  the  literature  searc  h are  presented 
in  the  appendix. 
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2.0  M ISSION  A N A 1 .Y  SIS 


A mission  analysis  was  conducted  to  establish  a framework  for  the 
definition  of  possible  simultaneous  sensor  combinations  and  their  subsequent 
evaluation.  The  new  VFAX  strike  fighter  aircraft,  the  Navy  F- 18  aircraft, 
was  selected  as  the  primary  vehicle  for  the  mission  analysis.  The  approa<  h 
was  to  determine  those  mission  phases  and  associated  operator  tasks  wherein 
simultaneous  sensor  presentation  techniques  might  be  employed  to  improve 
pilot  performance  and/or  reduce  operator  task  loading.  This  section 
includes : 

1.  Discussion  of  an  F-18  aircraft  air-to-ground  combat  mission 
(interdiction) , 

2.  Description  of  the  F-18  avionics  suite  including  sensors,  displays, 
and  ordnance,  and 

3.  Description  of  a functional  block  diagram  of  the  avionics  subsys- 
tems for  air-to-ground  combat  modes. 

2.  1 MISSION  SELECTION 

The  Navy  F-  18  aircraft  is  a single  place,  twin  engine,  high  perfor- 
mance fig hte r / attack  aircraft.  Its  size  falls  between  the  A- 7 and  F-4  and  is 
intended  to  eventually  replace  these  aircraft.  The  avionics  system  is 
designed  around  one-man  operation. 

The  F-18  is  designed  for  both  aircraft  carrier  and  conventional  land 
takeoff  and  landing.  The  aircraft  carrier  launch/ recovery  cycle  is  on  the 
order  of  116  minutes,  which  represents  the  unrefueled  endurance  of  con- 
temporary aircraft.  Marine  Corps  missions  from  deployed  shore  bases  will 
be  shorter  in  duration,  on  the  order  of  85  minutes,  because  distances  from 
the  launch/ recove ry  point  to  the  combat  zone  are  shorter  than  for  carrier 
deployed  aircraft.  Maximum  aircraft  speed  will  be  about  Mach  1.  6 with  a 
combat  speed  of  Mach  0.  9. 

Six  primary  missions  are  postulated  for  the  F-18  aircraft:  fighter 

escort,  self-escorted  interdiction,  combat  air  patrol,  close  air  support, 
deck-launched  interceptor,  and  ferry  missions.  The  typical  Navy  mission  is 
assumed  to  be  air-to-air  combat  at  10,000  foot  altitude;  the  typical  USMC 
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mission  is  assumed  to  be  close  air  support/ground  attack  at  sea  level 
altitudes.  For  this  analysis,  the  interdiction  mission  was  selected  for 
detailed  analysis  because  it  includes  mission  phases  and  operator  tasks  which 
require  the  simultaneous  utilization  of  a variety  of  sensors.  Thus,  it  is  dur- 
ing these  mission  phases  that  mission  performance  is  most  likely  to  be 
improved  by  optimized  sensor  display  techniques,  including  simultaneous 
sensor  presentations  on  a single  display. 

The  interdiction  mission  involves  ground  attack  against  prebriefed 
fixed  targets  whose  coordinates  are  known  and  against  targets  of  opportunity 
(stationary  or  moving)  in  a prebriefed  target-rich  area.  Both  cases  require 
precise  navigation  and  depend  heavily  on  penetration  tactics  and  self-contained 
target  acquisition  and  weapon  delivery.  Thus,  in  both  forms  of  interdiction 
missions,  the  single  seat  attack  operator  must  deal  with  a high  level  of  task 
loading,  stemming  from  the  simultaneous  requirements  of  survivability  and 
successful  target  acquisition  and  weapon  delivery.  The  task  loading  is  further 
intensified  in  night  operations.  The  goal  of  this  study,  then,  is  to  define  those 
simultaneous  sensor  presentations  (using  techniques  which  can  be  practicably 
implemented)  which  have  the  potential  to  improve  survivability,  increase  the 
probability  of  successful  target  acquisition  and  weapon  delivery,  and  achieve 
all  of  these  objectives  with  a reasonable  operator  task  loading. 

The  interdiction  mission  includes  12  mission  phases  as  illustrated  in 
Figure  1: 

1.  Takeoff 

2.  Climb 

3.  Cruise 

4.  Descent  to  sea  level 

5.  Run-in  to  target 

6.  Air-to-ground  combat 

7.  Run-out 

8.  Climb 

9.  Cruise 

10.  Descent  to  sea  level 

11.  Reserve  (sea  level  loiter) 

12.  Land 
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0 10  20  30  40 

MISSION  TIME,  MINUTES 

Figure  1.  Interdiction  mi s sion  profile . 


Typical  variations  in  the  run-in  to  target  and  air-to-ground  combat 
phases  are  shown  in  Figure  2.  In  both  Figure  a and  b,  preliminary  target 
acquisition  is  typically  achieved  at  ranges  of  at  least  15  n.  mi.  which  can  be 
greatly  extended  depending  on  prebriefing,  targetting  information,  and  the 
performance  characteristics  of  the  long  range  acquisition  sensors  (typically 
radar).  Target  coordinates  are  entered  in  the  mission  computer  and  the  air- 
craft descends  for  low-altitude  ingress  (~T00  feet  for  day  missions,  ~500  ft 
for  night).  At  pop-up,  the  target  must  be  reacquired.  In  Figure  2a  pop-up 
occurs  at  stand-off  ranges  of  about  50,000  feet,  allowing  more  time  for  target 
acquisition  and  guided  weapon  delivery  but  also  increasing  the  period  of  vul- 
nerability to  air  defense  sites  (fixed  or  mobile).  In  Figure  2b,  pop-up  is 
delayed  until  about  10,000  feet  target  range,  allowing  a minimum  of  time  for 
acquisition  and  strike,  but  also  minimizing  exposure  time  to  air  defense. 

The  air-to-ground  mission  phase  profile  shown  in  Figure  2a  might  be 
represented  by  an  F-  18  equipped  with  a high  performance  FLIR  (compatible 
with  target  detection  ranges  in  excess  of  50,000  feet),  a laser  spot  tracker  for 
cuing  in  a hunter-killer  mode,  a laser  designator  for  designation  of  targets 
for  laser-guided  bombs  or  missiles  (Bulldog  or  Maverick)  or  hand-off  to 
another  strike  aircraft,  and  a laser  ranger  for  computation  of  accurate  wea- 
pon release  conditions.  This  kind  of  sensor  suite  can  be  described  as  "TRAM- 
like".  The  radar  target  acquisition  and  weapon  release  modes  would  be  con- 
sidered back-ups  to  the  TRAM  (Target  Recognition/ Attack  Multisensor;  A-6E 
and  A-7E  versions). 
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DM44  371  ? 


CRUISE 

OUT 


PRELIMINARY  TARGET  ACQUISITION 

(RADAR  OR  RWR;  VERIFICATION  BY  OTHER  SENSORS  OR  DATA) 


Figure  Z.  Typical  interdiction  run-in  to  target  and 
air-to-ground  combat  phases. 
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Pop-up  at  stand-off  ranges  (Figure  2a)  could  be  used  in  attacking  air 
defense  sites.  The  targets  would  be  initially  detected  with  emitter  locators; 
target  coordinates  would  be  entered  in  the  INS/computer  prior  to  descent  and 
run-in;  and  pop-up,  reacquisition  and  weapon  delivery  would  be  accomplished 
at  6 to  7 n.  mi. 

A variation  of  this  maneuver  is  being  explored  in  the  Improved  Wild 
Weasel  program,  using  HR  Maverick  missiles.  The  corresponding  mission 
profile  is  shown  in  Figure  2b.  The  F-  18  pops  up  as  before  according  to  the 
emitter  locator  data,  but  at  the  last  possible  moment.  The  F-18  then  reac- 
quires the  target  and  hands-off  to  a launch  and  leave  missile  such  as  a TV 
or  IIR  Maverick,  followed  by  pop  down  and  egress.  The  target  reacquisition 
may  be  assisted  by  use  of  the  F LIR  in  the  sensor  suite.  In  this  mode  of  last- 
minute  pop-up,  the  laser  spot  tracker  and  laser  designator /ranger  have 
lesser  importance,  compared  to  the  stand-off  pop-up  case. 

The  utilization  of  the  profiles  described  in  Figure  2 depend  on  the 
weapon  and  flight  mix.  The  run-in  to  target  and  air-to-ground  combat  mis- 
sion phases  described  above  hold  the  greatest  potential  for  simultaneous 
sensor  presentation  techniques  since  they  require  that: 

1.  A mix  of  defensive  and  offensive  sensors  be  utilized,  and 

2.  The  sensor  data  be  acquired,  processed  and  displayed  in  the 
most  meaningful  and  quickly  interpretable  manner  to  achieve 
mission  success. 

2.  2 AVIONICS  SUITE 

The  F-18  aircraft  will  carry  a sophisticated  avionics  suite,  as  illus- 
trated in  Figure  3,  including  navigation  systems,  communications  and  identi- 
fication systems,  electronic  warfare  systems,  an  airborne  weapon  control 
system,  and  a set  of  displays  and  controls.  Centered  on  two  central  digital 
computers,  the  avionics  equipment  is  tied  with  a multiplex  system  providing 
flexibility  for  altering  or  adding  new  equipment.  Key  elements  of  interest 
for  this  study  are  the  interdiction  mission  armament,  the  aircraft  sensors 
and  missile  sensors,  and  the  displays  which  provide  most  of  the  combat 
management  information  to  the  pilot. 
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HUD 


Multi-mode 

Display 


Up  Front  Control 


Horizontal 

Situation 

Display 


Electronic  Wartare 

• Radar/SAM  Warn. 

• Active  ECM 

• Decoy  Dispensers 


Controls  And  Displays 


Radar  Display 


Comm  And  Ident 

• UHF  (2) 

• IFF 

• IFF  CRYPTO 

• Secure  Voice 

• Data  Link 


Figure  3.  F-18  avionics  suite. 


2.  2.  1 Ordnance 

Air-to-air  weapons  and  SAMs  are  considered  to  be  the  major  threat 
for  the  air  superiority  mission.  Ground-to-air  AAA  and  SA-6/7/8/9  are  con- 
sidered the  major  threat  to  the  air-to-ground  strike  missions. 

The  postulated  ordnance  for  the  F-18  includes:  A1M-7F,  A1M-9L, 

Mk  82,  Mk  83,  Mk  84,  Bulldog,  Maverick  (AGM-65A,  B,  C,  D),  Walleye 
(TV  and  IR),  HARM,  CBU,  FAE  II,  rockets,  fire  bombs  and  20  mm  gun. 

The  armament  is  carried  on  nine  store  stations,  as  shown  in  Figure  4,  for 
a total  capability  of  nearly  14,  000  pounds.  The  interdiction  mission  is  shown 
with  the  carriage  of  two  Sidewinders  (close-in  missiles  on  each  wing  tip  for 
low  drag  and  optimum  sensor  look  angle),  an  M61,  20  mm  cannon  (mounted 
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TAB  1 ,l\  I I 18  SKNSOR  SUIT  F] 


BASELINE  SENSORS 

DESIR  ABl.F"  OPTIONAL  SENSORS 

MISSILE  SENSORS 

VISUAL  TARGET  ACQUISITION 
SYSTEM  (V T AS) 

• DAY  TV  TRACKER  (SIMILAR 
TO  NAVY  PAVEKNIFE  OR 
IMPROVED  WEAPON  DELIVERY 
SYSTEM) 

• HIGH  RESOLUTION 

TV  IDENTIFICATION  SYSTEM 
(SIMILAR  TO  TISEO) 

LASER  AIDED  LLLTV 

TV  MAVERICK 
TV  WALLEYE 

FLIR  (SIMILAR  TO  AtfE  OR 
A 7E  TRAM! 

• HIGH  RESOLUTION 

IR  IDENTIFICATION  SYSTEM. 
IR  WARNING  RECEIVER 

HR  MAVERICK 
IR  WALLEYE 

LASER  SPOT  TRACKER 

LASER  AIDED  SEARCH  AND 
TRACK  (LAST)  POD 

TOA/DME 

LASER  DESIGNATOR  RANGER 

LASER  SEMIACTIVE 
(BULLDOG.  MAVERICK 
AND  LASER  GUIDED 
BOMBSI 

RADAR  WARNING  RECEIVER 

LASER  WARNING  RECEIVER 

AIR  TO  GROUND  RADAR 

• REAL  BE  AM  GROUND  MAP 

• DOPPLER  BEAM  SHARPENED 
SECTOR  OR  PATCH 

• GROUND  MOVING  TARGET 
1 N DIC  ATI  ON,  TRACK 

• FIXED  TARGET  TRACK 

• A-G  RANGING 

• TERRAIN  AVOIDANCE 

• SEA  SURFACE  MAPPING 

• SYNTHETIC  APERTURE 
RADAR  STRIP  MAP  AND 
TELESCOPE  TRACK 

• BEACON 

• TERRAIN  FOLLOWING 

• SEABORNE  MOVING 
TARGET  MAPPING 

AIR  TO  AIR  RADAR 

• PULSE  SEARCH 

• VELOCITY  SEARCH 

• RANGE  WHILE  SEARCH 

• TRACK  WHILE  SCAN 

• SINGLE  TARGET  TRACK 

• NONCOOPERATIVE 
TARGET  RECOGNITION 

• AIR  COMBAT  MODES 

SEMIACTIVE,  ACTIVE 
RADAR,  CW  AND  PD 

DATA  LINK 

• JOINT  TACTICAL  INFORMATION 
DISTRIBUTION  SYSTEM 

A PRIORI  SENSOR  DATA 


2.  2. 3 Displays 


The  integrated  display  set  includes: 

• A head-up  display  (ITUD)  which  superimposes  symbology  and 
alphanumeries  on  the  pilot's  view  of  the  outside  world,  providing 
weapon-aiming  information  as  well  as  essential  navigation  and 
primary  flight  instrumentation. 

• Two  multipurpose  display  indicators  (MDI),  one  on  the  right  and 
one  on  the  left,  to  provide  a radar  display  and  a multimode  dis- 
play for  control  of  armament,  mode  selection,  built-in-test,  and 
display  of  EO  weapon  sensors, 

• A moving  map  display  with  the  EW  information  presented  over  the 
map,  and 

• An  up-front  control  panel  between  the  two  MDls  for  selection  and 
control  of  communication  and  identification. 

2.  3 AVIONICS  INTERFACE  FOR  AIR-TO-GROUND  COMBAT 

Based  on  the  interdiction  mission  defined  in  Section  2.  1 and  the  F- IS 
avionics  suite  described  in  Section  2.  2,  the  interfacing  of  key  avionics  utilised 
in  the  target  acquisition  and  weapon  delivery  phases  can  be  examined.  The 
avionics  functional  block  diagram  is  illustrated  in  Figure  5,  configured  from 
the  baseline  F-18  sensor  suite  listed  in  Table  1,  and  compatible  with  the  sys- 
tem requirements  of  Figure  2.  This  configuration  includes: 

• A multimode  radar  system  such  as  the  F-18  radar, 

• A TRAM-like  system  including  the  F LIR  and  laser  designator/ 
ranger/spot  tracker, 

• Weapon  system  including  the  missile  imaging  sensors, 

• Laser/radar  warning  receivers,  IFF  threat  identification,  and 
data-linked  threat  data, 

• Navigation  systems  including  INS,  TACAN,  and  GPS, 

• General  processing  equipment  such  as  the  central  processing 
unit  (CPU)  and  the  weapon  release  computer  system  (WRCS), 

• Dedicated  computers  including  the  air  data  computer,  processors 
associated  with  the  TRAM-like  system  and  the  radar  system  for 
video  processing  and  coordinate  transform,  and  the  video  cor- 
relation hand-off  and  boresight  unit  which  interfaces  the  TRAM 
and  missile  imaging  sensors. 
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Figure  5.  Functional  block  diagram  of  general 
interdiction  mission  avionics. 


Figure  6 shows  the  utilization  of  the  various  avionics  subsystems 
during  cruise-in  (navigation),  during  preliminary  target  acquisition,  during 
target  reacquisition  at  a stand-off  range  pop-up,  and  during  a minimum 
range  pop-up.  In  all  these  cases,  it  is  seen  that  a multiplicity  of  sensors 
are  sampling  various  sets  of  the  navigation,  target  acquisition  and  weapon 
release  data.  Furthermore,  these  various  sets  of  data  are  being  utilized 
in  some  manner  by  all  the  mission  displays. 

Utilization  of  the  various  sets  of  data  is  essential  for  accomplishing 
the  mission.  The  ability  to  sample  multi-attributes  of  a target  was  demon- 
strated in  Viet  Nam  to  be  a definite  asset  for  rapid,  high  probability  of  target 
acquisition  (private  communication,  Col.  J.  Krause,  USAF).  Yet,  human 
factors  studies  (see  literature  survey,  Appendix  A)  have  shown  that  an 
observer's  performance  is  degraded,  especially  with  respect  to  time,  if  an 
observer  must  acquire  and  assimilate  the  multi-attribute  data  from  several 
displays.  These  factors  bring  us  to  the  key  issue  of  this  study,  to  examine 
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a.  CRUISE-IN  (NAVIGATION) 


EO  ft  CUING  SENSOR  SUITE 


THAM  LIKE  SYSTEM 

• FUR  (HOT  SPOT  DETECTION) 

• SERVO  ELECTRONICS 

• PROCESSOR  ELECTRONICS 


RADAR 


F 18  RADAR  LIKE  SYSTEM 

• MULTIMODE  A G. 

A A RADAR 

• PROGRAMMABLE  SIGNAL 

PROCESSOR 

• RADAR  DATA  PROCESSOR 


CENTRAL  PROCESSING  UNIT 


DIGITAL  MISSION 
COMPUTERS 

WEAPON  RELEASE 
COMPUTER  SYSTEM 

AIR  DATA  COMPUTER 


APRIORI  OATA 


DISPLAYS/CONTROLS  SYSTEM 


• HEAD  UP  DISPLAY 

• RADAR  DISPLAY 

• EO  WEAPON  A C SYSTEMS 

MANAGEMENT  DISPLAY 

• MOVING  MAP  DISPLAY 

• SCAN  CONVERTER 

• SYMBOL  GENERATOR 

• MICROPROCESSOR  CONTROL 


• MAPS 

• STORED  IMAGERY 

• THREAT  DATA  ETC 


NAVIGATION  AIDS 


• INS 

• TACAN 

• DATA  LINK 


WARNING  INFORMATION 


• RADAR  SAM  WARNING 
RECEIVER 

• LASER  WARNING  RECEIVER 

• IFF 

• DATA  LINK 


b.  PRELIMINARY  TARGET  ACQUISITION 

Figure  6.  Avionics  functional  block  diagram 
per  mission  phase.  (Sheet  1 of  2) 
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D7344  3716 


c.  STAND-OFF  RANGF  POP-DP  T ARC. FT  ACQUISITION 


d.  MINIMUM  RANGF  POP-UP  TARGFT  ACQUISITION 

Figure  6.  Avionics  functional  block  diagram 
per  mission  phase.  (Sheet  2 of  2) 
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techniques  of  simultaneously  presenting  the  various  essential  sets  of  data 
on  a single  display  in  order  to  obtain  the  target  acquisition  improvement 
associated  with  multi-attributes  while  avoiding  an  increase  in  operator  task 
loading. 

This  issue  is  addressed  here  in  the  following  manner.  In  each  of  the 
mission  phases,  a fundamental  sensor  or  data  set  is  selected,  and  all  other 
simultaneously  obtained  sensor  information  or  data  are  considered  supple- 
mentary. In  practice,  this  means  that  the  fundamental  sensor  information  or 
data  set  will  be  displayed  on  its  usual  monitor,  and  the  supplementary  data 
will  be  overlayeri  as  raw,  processed,  or  symbolized  data.  The  fundamental 
data  set  may  sometimes  be  radar  (Figure  6 a and  b),  and  sometimes  EO 
(Figure  6 b and  c).  Thus,  all  F- 18  video  displays  will  have  a role  in  dis- 
playing the  fundamental  sensor  data  and  its  supplementary  overlay.  Con- 
sistent with  the  F-  18  design  and  controls/displays  layout,  several  displays 
are  retained  in  this  analysis.  However,  it  is  also  possible  to  consider  a 
single,  multipurpose  optimized  display  after  defining  the  requirements,  as 
discussed  further  in  Section  4.  0. 

In  the  following  sections,  the  particular  capabilities  of  the  baseline 
and  desirable  sensors  in  the  F-18  avionics  suite  are  discussed.  This  dis- 
cussion allows  the  fundamental  data  sets  to  be  defined  for  each  critical 
mission  phase.  Once  this  is  accomplished,  the  optimal  methods  of  presenting 
the  fundamental  or  primary  data  and  the  supplementary  data  are  then 
addressed.  The  resulting  candidate  simultaneous  sensor  presentations  which 
appear  most  promising  are  defined,  accompanied  by  representative  simulated 
imagery.  The  simulated  simultaneous  presentations  allow  a preliminary 
assessment  to  be  made  of  the  utility  of  the  combined  sensors  or  data  sets 
to  the  pilot. 
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3.0  METHODOLOGY  FOR  SELECTING.  MULTISENSOR  CANDIDATES 


To  evaluate  the  potential  utility  and  practicality  of  multisensor  fusion 
on  a single  display,  the  following  framework  was  established  as  described 
in  Section  2.  0; 


1.  An  interdiction  mission  for  the  F-18  aircraft  was  selected 
in  which  target  acquisition  time,  operator  task  loading,  and 
aircraft  survivability  are  critical  factors. 

2.  The  F-18  avionics  suite  was  defined  including  the  baseline 
sensor  suite,  optional  sensors,  and  missile  sensors. 

3.  The  present  interfacing  of  the  avionics  elements  to  provide  the 
pilot  of  a single  seat  A/C  with  the  sensor  information  needed 
for  air-to-ground  combat  was  examined. 


This  section  describes  the  methodology  by  which  multisensor 
candidates  for  the  interdiction  mission  were  selected  from  the  F-18  sensor 
suite  for  simultaneous  presentation  on  a single  display.  In  all  cases,  the 
goal  was  to  ease  the  operator's  task  load  or  to  increase  the  probability  of 
successful  task  performance  in  shorter  time  or  at  longer  range.  The 
resultant  baseline  sensor  candidates  are  summarized  in  Table  2 as  a func- 
tion of  mission  phase. 

The  selection  procedure  was  sequenced  as  follows: 

1.  The  general  capabilities  of  the  baseline  and  optional  sensors  in 
the  F-18  avionics  suite  were  examined. 

2.  The  types  of  a priori  or  data-linked  information  that  may  be 
available  for  display  throughout  the  interdiction  mission  were 
determined. 

3.  The  operator  tasks  associated  with  the  information  provided 
by  each  sensor  or  data  source  were  defined. 

4.  Rased  on  operator  tasks  per  mission  phase,  information 
requirements,  and  sensor  capabilities,  those  sensors  or  data 
sources  which  could  be  utilized  during  each  phase  of  the  inter- 
diction mission  were  determined. 

5.  Sensors  which  could  be  used  within  the  target  acquisition  phases 
of  the  interdiction  mission  were  further  subdivided  according 

to  type  of  target  — fixed  or  mobile  targets,  moving  targets,  and 
emitte  r s . 


As  a result  of  these  five  considerations,  several  logical  groupings 
of  sensors  emerge  for  possible  combination,  based  on  similarities  in 
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TABLE  2.  SENSOR  CANDIDATES  FOR  COMBINATION  AS 
A FUNCTION  OF  MISSION  PHASE 


••  HIGHLY  DESIRABLE  OPTIONS  INCLUDED  WITH 
BASELINE  AS  DEFINED  IN  TABLE  3-1 

operational  range  as  well  as  the  existence  of  multiple  complementary  signa- 
tures for  a single  target  in  diverse  regions  of  the  electromagnetic  spectrum. 

3.  1 SENSOR  CAPABILITIES 

A baseline  sensor  suite  for  the  F-18  aircraft  was  identified  in 
Section  2.0,  as  well  as  several  desirable  but  optional  sensors.  To  deter- 
mine which  of  these  sensors  are  compatible  for  simultaneous  data 
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presentation  on  a single  display,  the  operational  characteristics  and 
capabilities  of  each  sensor  have  been  examined  in  terms  of: 


• Type  of  information  which  the  sensor  provides, 

• Typical  format  in  which  that  information  is  displayed  to  the 
ope  rato  r , 

• Field  of  regard  of  the  sensor,  i,  e,  , total  angular  coverage 
within  which  the  sensor  field  of  view  can  be  positioned  (for 
example,  gimbal  limits  of  the  sensor  platform), 

• Field  of  view  of  the  sensor,  i.  e.  , angular  coverage  of  the 
sensor  within  one  update  or  frame  time, 

• Sensor  resolution, 

• Update  rate,  i.  e.  , time  during  which  information  is  acquired 
from  full  field  of  view  and  processed  for  display, 

• Dynamic  range,  i.e.  , ratio  of  maximum  to  minimum  signal 
which  can  be  detected  or  processed  by  the  sensor  under  normal 
operating  conditions, 

• Operational  range,  i.e,,  range  between  target  and  sensor  at 
which  specific  operator  tasks  can  be  accomplished  via  the 
sensor  display  such  as  target  detection,  recognition,  etc., 

or  specific  sensor  /processor  tasks  can  be  accomplished  such 
as  A-G  GMI’I  or  A-A  Track- While- Scan , etc. 

Characteristics  of  the  information  provided  by  the  F- 18  baseline 
and  optional  sensors  are  summarized  in  this  section.  Typical  values  of 
key  sensor  parameters  are  listed  in  Table  3. 

3.  1,  1 Television  (TV) 

Airborne  TV  sensors  have  been  developed  for: 

1.  Daytime  long  range  target  identification,  precision  tracking 
(using  gated  video  trackers),  and  missile  guidance;  and 

2.  Ixjw  light  level  observation  of  terrain  for  purposes  of  flight 
control,  general  orientation,  target  detection,  acquisition, 
tracking,  identification  and  weapon  delivery. 

TV  sensors  detect  luminance  contrast  within  a scene  and  provide 
a/.imuth- elevation  video  imagery.  At  low  light  levels,  TV  performance  is 
severely  limited  compared  to  the  infrared  sensor.  The  low  light  level  TV 
(LLLTV)  with  image  intensifier  stage  was  developed  for  nighttime  operation 
and  functions  down  to  about  1/4  moon  illumination.  Below  this  level,  an 
illuminator-aided  TV  has  been  explored  to  improve  nighttime  performance 
by  using  pulsed  lasers  for  illumination  of  the  object  space. 
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TV  scan  standards  include  525,  645,  787,  875,  945  and  1024-line 
rasters  with  60  fields  per  second,  2:1  interlace  and  a 3:4  aspect  ratio. 

Missile  TVs  (Maverick,  Walleye,  Condor)  are  525-line  square  rasters  and 
are  typically  lower  resolution  sensors  with  comparatively  less  performance 
than  target  acquisition  TVs.  The  effective  resolution  of  TV  sensors,  in 
general,  is  considerably  lower  than  that  implied  by  the  number  of  raster 
lines.  Typical  resolution  for  525-line  rasters  is  about  300  lines,  and  600  to 
800  lines  for  1024-line  TVs.  The  effective  resolution  of  LLLTVs  varies 
with  light  level  and  can  be  as  high  as  300  to  800  lines  under  bright  moonlight 
conditions  and  as  low  as  100  to  200  lines  under  very  dark  conditions. 

As  with  most  types  of  pictorial  information,  a wide  dynamic  range 
is  preferred.  For  this  reason,  automatic  contrast  and  brightness  control 
are  typically  a part  of  the  overall  system.  TV  is  generally  most  useful 
at  close  to  medium  target  observation  ranges  and  gives  a visual  impression 
to  the  observer  which  is  most  like  the  appearance  of  a direct  visual  scene 
under  ordinary  illumination  conditions.  The  minimum  number  of  discernible 
gray  levels  in  the  system  must  be  approximately  10  to  achieve  imagery 
without  objectionable  quantization  effects.  TV  systems  are  among  the  most 
visually  noise-free  of  all  the  sensors  available. 

TV  sensors  in  an  attack  aircraft  are  gimballed  and  stabilized;  lower 
hemisphere  coverage  can  be  readily  accomplished.  Video  tracker  capability 
and  slaving  to  other  tracking  sensors  are  typically  included  in  the  TV  system 
design. 

3.  1.2  Forward-Looking  Infrared  (FLIR) 

FLIR  sensors  detect  temperature  differences  within  a scene  and 
provide  a zimuth- el  ovation , video  imagery  with  shades  of  gray  similar  to 
TV  imagery.  Because  of  the  longer  wavelength  and  generally  lower  number 
of  scan  lines  (typically  less  than  500),  the  resolution  is  not  as  good  as  TV. 

The  system  also  may  operate  more  slowly  than  TV,  perhaps  at  15  to  20  frames 
per  second.  Recause  of  the  strong  infrared  emissions  of  vehicles,  machinery, 
etc.,  the  dynamic  range  of  FLIR  systems  must  be  large  to  accommodate 
these  emitters  as  well  as  a relatively  weak  background.  An  important  visual 
characteristic  is  that  the  strong  emitters  tend  to  stand  out  sharply  as  "hot 
spots''  which  may  tend  to  bloom  on  a display  without  automatic  gain  control. 
FLIR  systems  are  most  useful  at  short -to-medium  range,  and  10  or  more 
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gray  levels  are  generally  necessary  to  avoid  quantization  noise  as  with 
visual  TV.  Noise  levels  in  the  image  may  be  somewhat  higher,  and  scan 
conversion  and  Irame  storage  often  is  required  to  convert  the  data  to  a 
TV-type  format. 

FLIR  is  particularly  valuable  for  target  acquisition  and  weapon 
delivery  at  night.  Two  or  more  fields  of  view  (-20°  to  1°)  are  typically 
provided  for  search  and  precision  track /identification  capability. 

The  I-  LIR  sensor  is  gimballed  and  stabilized,  with  gimbal  ranges 
in  some  cases  extending  through  a lower  hemisphere  coverage  to  permit 
target  tracking  for  weapon  guidance  through  impact.  Advanced  FLIRs  will 
have  automatic  video  tracking  capability.  FLIR  systems  which  have  standard 
525  or  875-line  TV  raster  operation  can  be  used  with  conventional  TV 
trackers  where  FLIR/TV  combinations  are  employed. 

Missile  FLIRs  are  525-line  TV  compatible,  but  typically  have  lower 
performance  and  lower  effective  resolution  than  the  FLIR  in  the  aircraft 
target  acquisition  pod. 

3.  1,  3 IR  Search  and  Track 

These  IR  systems  use  a small  number  of  detectors  which  are  scanned 
rapidly  in  azimuth  and  elevation  to  detect  interceptors  and  anti-aircraft 
missiles  above  or  below  the  horizon.  In  practice,  they  can  be  positioned 
lor  nose  or  tail  warning.  The  output  signals  are  displayed  as  intensity- 
modulated  cell  traces  in  a C-scan  elevation  versus  azimuth  format  and 
require  scan  conversion  to  be  TV  format  compatible.  High  resolution 
detector  arrays  can  be  used  such  that  the  detected  threat  could  be  tracked 
and  imaged  for  identification  at  closer  ranges. 

3.  1.4  Laser  Spot  Tracker /Designator /Ranger 

Within  the  interdiction  mission,  the  laser  spot  tracker  can  be  used 
to  track  a ground  target  being  designated  by  an  AFAC  (airborne  forward 
air  controller)  and  to  provide  guidance  information  for  aircraft  strike.  The 
aircraft  can  utilize  its  own  laser  designator  equipment  to  designate  a target 
for  strike  by  itself  or  by  another  aircraft  when  equipped  with  laser-semiactive 
weapons  such  as  laser-guided  bombs.  The  laser  ranger  can  be  used  to  pro- 
vide accurate  range  data  for  navigation  update  and  for  weapon  delivery 
computations. 
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3.  1.5  Laser/Radar  Warning  Receiver  (L/RWR) 


The  RWR  is  in  general  use  on  military  aircraft  to  provide  detection, 
bearing,  identification,  mode  of  operation  and  prioritization  of  threats  such 
as  SAMs  (including  missile  guidance  activity /missile  launch),  AI,  AAA,  and 
ground  control  intercept,  acquisition,  and  height  finder  radars.  The  LWR 
provides  detection,  direction  of  arrival,  and  signature  characteristics  of 
threat  lasers.  Each  L/RWR  threat  is  identified  by  an  A/N  symbol  on  the 
360  degree  azimuth  - range  display  and  can  be  accompanied  by  an  audio  tone 
warning.  Hand  off  of  the  video  of  selected  threats  to  other  aircraft  systems 
is  provided  for  jamming  or  weapon  delivery. 

3.1.6  Rada  r 

Radar  modes  described  here  are  intended  to  allow  the  optimum  level 
of  air-to-ground  attack  and  air-to-air  combat  for  operation  in  a single- 
place, high  performance  aircraft.  To  minimize  required  pilot  action,  the 
radar  should  provide  automatic  mode  and  parameter  management  to  the 
maximum  practical  extent  with  manual  override.  It  is  highly  desirable  that 
the  radar  feature  programmable  signal  and  radar  data  processing.  In  this 
manner,  mode  logic  and  waveform  parameters  can  be  refined  as  field 
experience  accumulates,  and  radar  data  can  be  translated  into  useful  pilot 
data  with  total  software  control. 

Air  -to-  Ground 

The  radar's  air-to-ground  capability  includes  a complement  of  modes 
which  enable  the  pilot  to  1)  navigate  to  a wide  class  of  checkpoints  and  tar- 
gets, with  LORAN  precision  and  at  all  penetration  altitudes,  and  2)  to 
detect,  acquire  and  track  stationary  and  moving  ground  and  sea  targets. 

• The  real  beam  ground  map  (RRGM)  mode  provides  PPI  mapping 
of  large  ground  areas  (out  to  160  nmi  range  and  120  degree 
azimuth  sector)  for  general  navigation  use.  PPI  radar  sensors 
have  visual  display  characteristics  much  different  from  tele- 
vision. The  images  have  little  gray-level  quality,  and  objects 
tend  to  be  visible  as  points  and  edges  rather  than  broad  filled-in 
areas.  The  scanning  format  is  polar,  in  azimuth  and  elevation, 
thus  requiring  scan  conversion  for  display  on  a rectangular 
scanned  display.  The  digital  scan  conversion  system  must  be 
sufficiently  flexible  to  accommodate  various  ranges  and  sectors 
of  the  poiar  scan  for  conversion  to  rectangular  format.  The 
radar  also  has  a large  dynamic  range,  and  "glints''  from  strong 
targets  in  a low-detail  background  are  common.  Noise  in  the 
form  of  clutter  and  object  scintillation  is  also  common. 
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Doppler  beam  sharpening  (DBS)  provides  a higher  azimuth 
resolution  ground  map  with  reduced  range  (40  nmi)  and  azimuth 
coverage  and  increased  processing  time.  Typically,  the  radar 
is  initially  in  a RBGM  mode  and  the  pilot  designates  a point 
of  interest  with  a cursor.  The  expanded  DBS  sector,  45  degrees 
azimuth  coverage  and  up  to  20  nmi  mapped  range  interval,  then 
appears  centered  in  range  and  azimuth  about  the  designated 
position.  Designation  of  a point  within  this  sector  results  in  a 
further  expansion  called  the  DBS  patch  mode.  A typical  DBS 
patch  is  4 x 4 nmi  at  20  nmi  range,  centered  in  range  and  azi- 
muth about  the  designated  position.  The  map  formed  from  the 
previous  scan  is  displayed  until  the  present  scan  is  complete 
resulting  in  sequential  snapshots. 

In  the  side-looking  synthetic  aperture  radar  (SAR)  mode,  the 
antenna  is  positioned  at  a fixed  angle  to  the  side  of  the  aircraft 
and  the  forward  motion  of  the  aircraft  provides  the  azimuth  scan. 
Synthetic  array  processing  provides  very  high  resolution  strip 
maps  in  range  "windows”  varying  from  1/2  to  20  nmi  which  may 
be  positioned  along  the  range  sweep  from  near  0 to  over  150  nmi. 
A further  refinement  is  the  tracking  telescope  mode  which  pro- 
vides discrete  frames  of  synthetically  processed  radar  video 
around  a designated  ground  point.  SAR  images  tend  to  have 
many  "glints"  due  to  reflections  from  points,  corners  and  edges 
of  man-made  objects,  resulting  in  an  "outlining"  or  edge 
enhancing  effect.  The  edge  enhanced  nature  of  SAR  pictures 
may  be  extremely  useful  for  frame  superposition.  SAR  pic- 
tures tend  to  have  a large  dynamic  range;  thus  some  type  of 
dynamic  range  compression  or  stretching  is  often  useful  before 
presentation  to  a human  observer.  The  system  operates  at 
several  hundred  scan  lines,  but  the  frame  rate  is  very  slow 
(can  be  one  or  more  seconds  per  fram'D  due  to  the  complicated 
processing.  The  SAR  images  are  useful  primarily  at  long  to 
medium  ranges,  and  scan  conversion  is  generally  necessary  for 
simultaneous  display  with  other  sensors.  Frame  storage  is  also 
required  due  to  the  relatively  slow  repetition  rate  of  the  new 
frames. 

Detection,  acquisition  and  tracking  of  both  fixed  and  moving 
ground  targets  is  provided.  Ground  moving  target  indication 
(GMTI)  symbology  can  be  overlaid  on  the  video  from  each  of 
the  ground  map  modes  — RBGM,  DBS  or  SAR.  Ground  moving 
target  track  (GM  TT ) and  fixed  target  track  ( FTT ) a re  displayed 
as  symbols  positioned  at  the  target  range  and  azimuth  on  a 
B-scan  format. 

The  beacon  mode  allows  for  detection  of  ground-based  beacons 
and  the  display  of  aj-  ropriate  symbology  overlaid  on  ground 
map  video. 

Air-to-ground  ranging  provides  slant  range  to  the  ground  along 
the  antenna  boresight  line  for  discrete  hard  targets  or  diffuse 
terrain,  and  may  be  precisely  pointed  to  aid  in  A-G  weapon 
del  ive  ry. 


• In  the  terrain  following  iTF)  mode,  the  radar  provides  data  to 
generate  an  elevation  versus  range  display  in  the  forward  direc- 
tion. This  elevation  profile  of  the  terrain  ahead  of  the  aircraft, 
combined  with  a safe  maneuver  template  overlaid  on  the  display, 
allows  the  aircraft  to  be  safely  flown  at  low  altitudes. 

• The  terrain  avoidance  (TA)  mode  is  generally  used  in  low  alti- 
tude flight  to  determine  optimal  horizontal  maneuvers  such  that 
exposure  to  detection,  tracking  and  countermeasures  can  be 
minimized.  Typically,  the  video  is  level  sliced  such  that  only 
terrain  or  man-made  objects  protruding  above  the  selected 
clearance  plane  are  displayed. 

• A C-scan  azimuth  versus  elevation  radar  mode  could  also  be 
incorporated  in  the  F-18  radar.  Radar  returns  could  be  pro- 
cessed such  that  returns  within  a discrete  range  interval  would 
be  displayed  as  a single  contour.  Coded  shading  of  these  con- 
tours would  result  in  a terrain  clearance  display. 


Air-to- Air 

If  the  F-18  is  attacked  by  an  interceptor  during  an  interdiction 
mission,  it  will  utilize  its  "self-escort'  capability  provided  by  the  large 
detection  coverage  of  its  sensors  and  the  aircraft's  ability  to  carry  and 
effectively  use  the  best  air-to-air  weapons  for  the  particular  threat  environ- 
ment. The  threat  ranges  from  AAA,  surface-to-air  missiles,  and  day 
fighters  in  the  limited  war  environment,  to  all-weather  and  supersonic 
cruise  interceptors  equipped  with  medium-range  standoff  missiles  in 
escalated  conflicts.  To  effectively  counter  these  threats,  the  radar  includes 
several  high  performance  air-to-air  search,  acquisition,  and  track  modes. 

The  pulse  search,  velocity  Search,  and  range  - while -search  modes 
provide  all -aspect  detection  capability  of  both  opening  and  closing  airborne 
targets  for  both  look-up  and  look-down  encounter  conditions.  The  single 
target  track  mode  provides  tracking  of  a single  target,  maintenance  of  this 
capability  in  an  ECM  environment,  and  continuous  wave  or  pulse  doppler 
illumination  for  missile  launch  and  flight.  The  track- while-scan  mode 
provides  simultaneous  tracking  and  prioritization  of  1Z  airborne  targets 
and  a display  of  8.  Three  air  combat  maneuvering  modes  are  also  incor- 
porated and  optimized  for  short  range,  heads-up  air-to-air  engagement. 
Modes  can  be  selected  manually  by  the  pilot  or  commanded  automatically 
by  the  mission  computer. 
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3,  1,7  Microwave  Kadiometry  (MICRAD) 


The  development  of  MICRAD  sensors  for  passive,  all-weather 
navigation  and  landing,  as  well  as  possible  target  acquisition,  is  presently 
receiving  considerable  interest.  MICRAD  sensors,  which  typically  operate 
in  atmospheric  windows  at  33  or  GH/,  depend  primarily  on  differences 
in  emissivity  to  image  a scene  and  can  operate  in  clouds,  rain,  smoke  and 
snow  cover.  Images  are  the  same  day  or  night  and  there  are  no  day/night 
inversions  or  dead  times.  Resolution  and  detection  ranges  are  strongly 
dependent  on  the  S/N  ratio  which  in  turn  depends  on  target/background 
contrast,  system  noise,  integration  time,  and  other  target,  antenna  and 
receiver  characteristics.  Although  the  MICRAD  sensor  is  limited  in 
range  and  resolution,  it  would  provide  passive,  all-weather  ground  mapping 
and  possible  air-to-air  threat  detection.  For  the  air-to-ground  target 
acquisition  and  navigation  update  tasks  of  interest  in  an  interdiction  mission, 
the  development  of  a forward-looking  MICRAD  sensor  capable  of  rapid 
two-dimensional  scanning  ta  microwave  equivalent  of  the  FLIR)  with  narrow 
field  of  view  and  slewable  over  a wide  angle  could  be  desirable.  Automatic 
terrain  correlation  navigation  with  MICRAD  imagery  has  already  been 
demonstrated  using  real-time  in-flight  digital  processing.  ' 

3.1.8  Data  Link 

The  types  of  information  which  will  be  data  linked  to  the  aircraft 
include  the  following: 

• Missile  sensor  data  which  can  be  flight  information  displayed 
by  means  of  symbology. 

• Missile  sensor  imagery  such  as  that  obtained  from  the  TV  (or 
eventually  IR)  Walleye  and  Condor. 

• JTIDS  — The  Joint  Tactical  Information  Distribution  System  is 
an  advanced  digital  communications  link  used  to  distribute 
real-time  data  to  air  and  ground  elements  in  tactical  operations. 

• CPS  — The  Global  Positioning  System  presently  in  development 
will  allow  worldwide  electronic  grid  navigation  using  satellites 
for  increased  navigation  and  weapon  delivery  accuracy. 


R.  P.  Moore,  "Microwave  Radiom 'trie  All-Weather  Imaging  and 
Piloting  Techniques,"  AGARD  Conference  Proceedings  No.  148, 
Paper  8,  May  1974. 
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3.1.9  Effect  of  Weather 


The  sensor  operational  ranges  listed  in  Table  3 represent  those 
ranges  at  which  the  specified  operator  task  (target  detection,  recognition, 
etc . 1 can  be  accomplished  during  average  weather  conditions.  Typically, 
this  corresponds  to  a visibility  of  about  12  km.  hi  adverse  weather  such  as 
fog  or  rain,  those  ranges  are  reduced  depending  on  the  spectral  sensitivity 
of  the  individual  sensor.  Figure  7 illustrates  the  relative  attenuation  of  the 
electromagnetic  spectrum  produced  by  fog  and  rain  as  a function  of  wave- 
length in  microns  (or  frequency  in  hertz).  These  attenuation  values  can 
be  applied  to  1)  passive  sensor  systems,  such  as  the  FLIR  and  TV,  to 
determine  the  amount  by  which  the  target's  emitted  thermal  or  reflected 
solar  radiation  will  be  reduced  over  the  ta rget- sensor  range;  and  2)  active 
systems,  such  as  radar  and  laser  des  ignato  r / ranger  / spot  trackers,  to 
determine  one-way  or  two-way  range  attenuation.  M.ICRAD  systems  can  be 
passive  or  active. 

It  is  evident  from  Figure  7 that  the  sensor  least  affected  by  adverse 
weather  is  the  radar,  either  in  fog  or  rain.  In  the  case  of  fog  (and  clouds) 
the  TV  is  most  severely  affected  due  to  reflection,  while  the  FLIR  is  often 
able  to  ''see  through"  patches  of  fog,  clouds  and  smoke.  (On  a very  humid 
day  in  the  absence  of  clouds,  however,  the  IR  absorption  by  water  vapor 
may  reduce  FLIR  performance,  while  the  TV  is  virtually  unaffected.  ) 
Similar  to  the  relative  effects  of  fog  on  FLIR  and  TV  performance,  the 
10.6  p laser  penetrates  fog  (and  smoke)  much  more  effectively  than  1.06  p 
laser  radiation. 

In  the  case  of  rain,  FLIR  and  TV  are  similarly  affected,  although 
far  less  severely  than  by  fog.  MICRAD  sensors,  by  contrast,  function 
extremely  well  in  fog,  but  can  often  be  outpe rfo rmed  by  FLIR  or  TV  during 
rain.  Note  that  all  the  sensors  in  Figure  7 can  he  used  at  night  except  for 
the  TV,  unless  an  intensifier  stage  is  added  or  it  is  illumination-aided. 

3.2  A PRIORI  AND  DATA- LINKED  INFORMATION 

The  type  of  a priori  (prebriefed)  or  data-linked  information  that  may 
be  displayed  to  the  pilot  was  examined  by  considering  a representative 
interdiction  mission  from  planning  phase  through  launch,  enroute  to  target, 
attack,  egress,  and  return.  The  key  relationships  of  a ground  control 
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station  and  an  aircraft  were  examined  to  determine  what  type  of  information 
should  be  in  the  aircraft  data  base  at  launch  and  what  information  might  be 
communicated  to  or  from  the  aircraft  throughout  the  mission. 

3.2.  1 A Priori  Information  at  Take-Off 


Preparing  for  the  mission  consists  of  anticipating  the  sequential  use 
of  sensors  throughout  the  mission  and  selectively  extracting  key  data  for  dis- 
play from  the  master  data  base  to  assist  the  operator  in  anticipating  events 
and  recognizing  deviations  and  new  developments.  In  this  preparation,  the 
task  may  be  divided  into  two  major  areas  of  consideration  as  illustrated  in 
Figure  8: 

1.  I'he  geographic  swath  along  the  route  representing  area  covered 
by  the  primary  sensors  (20  to  50  nm  width),  and 

2.  The  adjacent  or  secondary  areas  which  contain  threats  and  sys- 
tems which  may  interact  with  the  aircraft  although  they  are  out 
of  the  aircraft's  sensor  range. 


Figure  8. 
areas 


Primary  and  secondary  threat 
for  interdiction  mission. 
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As  the  target  is  specified  and  the  route  and  tactics  are  selected,  the 
ground  data  base  is  processed  to  extract  the  map  data  and  the  defensive  order 
of  battle  that  will  be  encountered  along  the  route.  The  map  data  can  be 
selected  for  primary  and  alternate  routes  to  primary  and  alternate  targets 
to  assist  in  navigation  and  target  location/verification.  The  map  data  are 
stored  in  the  aircraft  data  base  and,  in  the  F-18,  are  utilized  during  the 
mission  to  provide  a moving  map  display  on  the  horizontal  situation  indicator. 

The  order  of  battle  includes  the  specific  known  radars  and  locations 
which  the  aircraft  will  encounter  moment  by  moment.  These  threat  data  are 
stored  in  the  aircraft  data  base  and  presented  at  the  appropriate  time  on 
the  horizontal  situation  or  vertical  situation  display.  Symbol  coding  is  used 
to  identify  the  threats  as  data  from  the  file  and  not  information  from  the  air- 
craft sensors.  This  stored  threat  information  must  also  be  correlated  with 
the  aircraft’s  EW  systems  so  that  as  signals  are  processed  and  displayed 
by  the  EW  system,  the  operator  may  visually  correlate  the  EW  system  data 
to  the  stored  threat  data.  In  like  manner,  the  expected  image  of  the  radar 
and  IR  sensors  (as  well  as  reconnaissance  photos)  at  key  locations  along  the 
route  are  stored  and  available  for  comparison  with  the  real  environment. 

This  thorough  preparation  and  processing  is  essential  to  the  success 
of  the  mission  since  most  targets  are  in  defended  areas  and  the  more  dense 
the  defensive  system,  the  more  valuable  the  target.  During  the  conflict  in 
Southeast  Asia,  it  was  not  uncommon  for  the  radar  warning  receivers  of 
fighter  aircraft  to  become  saturated  with  signals  (and  thus  of  limited  value) 
during  large  strikes  into  North  Viet  Nam.  However,  by  comparing  pre- 
stored threat  data  and  prestored  imagery  with  real-time  data,  the  pilot 
may  verify  threat  locations  or  he  may  use  these  data  to  suppress  display  of 
non-required  information  and  thus  quickly  recognize  new  or  unexpected 
threats. 

Although  the  aircraft  EW  systems  are  not  usually  programmed  to 
recognize  early  warning  tracking  radar,  it  would  be  useful  to  determine 
the  aircraft's  initial  detection  by  enemy  early  warning  radars  and  display 
this  information  to  the  operator.  Also,  this  information  can  be  entered  in 
the  tactical  information  file  via  JTIDs.  The  categories  of  a priori  data 
obtained  prior  to  launch,  stored  in  the  data  base,  and  available  for  display  to 
the  operator  throughout  the  mission  by  means  of  symbology,  graphics  and 
imagery  are  summarized  in  Table  4. 
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TABLE  4.  A PRIORI  DATA  STORED  IN  DATA  RASE 


Ca  rtog  ra  phy 
Location  of  SAM  Sites 
Airfields 

Location  of  AAA  Batteries 

Initial  Radar  Detection  Line 

Expected  Locations  of  1R 
Sources 

Major  Objects  from  Radar 
Imag  ing 

Launch  Range  of  SAM  Sites 


Secondary  Target  Route  Data 

Expected  Radar  Target  Image 

Areas  of  Enemy  Fighter 
Encounte  rs 

Free  /.ones 

Restricted  Flying  Areas 
Location  of  Friendly  Forces 
Weather 


3.2.2  Data- Linked  Information  In  Flight 

The  aircraft  will  interact  with  the  overall  Joint  Tactical  Information 
Distribution  System  (JTIDS).  Own  aircraft  position  is  automatically  trans- 
mitted into  the  JTIDS  network  as  well  as  new  tracks  from  the  aircraft  radar 
and  avionics  systems.  The  operator,  in  turn,  accesses  data  from  the  net- 
work, either  surface  control  or  other  aircraft,  to  update  his  threat  infor- 
mation, navigation  system,  etc.  Typical  information  which  may  be 
communicated  to  the  aircraft  for  display  during  the  mission  is  listed  in 
Table  5. 


TABLE  3.  DATA  LINK  INFORMATION  (CHANGE  DATA) 


Alerts  to  Attack  by  Enemy 
Fighte  r s 

Changes  to  SAM/AAA  Locations 

Activity  by  SAM  Site 

Time  and  location  of  Initial 
Radar  Detection 

Route  Changes 

Mission  Profile  Changes 

Change  to  Alternate  Target 


Movement  of  Targets 

Movement  of  Friendly  Forces 

Relocation  of  Recovery  Point 

New  IR  Sources 

Update  Expected  IR  Images 

Update  Expected  Radar 
Images 

Weather  Changes 
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The  JTIDS  processor  correlates  own  aircraft  track  against  a 
maximum  of  200  stored  tracks  including  friendly,  hostile  and  unknown  tracks, 
SAM/AAA  sites,  way  points  and  weather  data.  Typically,  own  aircraft 
position  is  shown  as  a symbol  on  the  center  of  the  display  and  up  to  40  tracks 
can  be  presented  up  to  the  limits  of  the  display.  The  display  center  can  be 
offset  to  anywhere  within  a 650  nm  radius.  The  operator  can  select  either 
hostile  SAM/AAA  and  hostile  airbases  or  hostile  aircraft  as  the  highest 
priority.  Typical  JTIDS  display  symbology  is  shown  in  Table  6.  These 
symbols  could  be  placed  on  a clutter-free  display,  accompanied  by  an 
electronically  generated  map  or  overlaid  on  video  such  as  a radar  ground 
map. 

3.3  USE  OF  SENSOR  INFORMATION  FOR  OPERATOR  TASKS 

The  sensors  on-board  the  strike  aircraft  provide  for  the  general 
functions  of: 

1.  Threat  detection, 

2.  Threat  defense, 

3.  Ground  target  attack,  and 

4.  Data  receiving. 

Sensors  for  threat  detection  include  air-to-air  radar,  radar/laser 
warning  receivers,  and  possibly  an  IR  search  and  warning  receiver.  The 
operator's  task  with  these  sensors  is  principally  monitoring  the  appropriate 
display,  attending  to  aural  warning  from  the  warning  receivers,  and  inter- 
pretation of  the  displayed  information. 

Threat  defense  sensors  include  air-to-air  radar,  IFF  transponder/ 
receiver,  TV,  and  terrain  following/terrain  avoidance  radar.  Air-to-air 
radar  would  be  used  by  the  operator  to  lock-on  and  attack  an  airborne 
threat.  The  IFF  transponder /receiver  and  high  resolution,  narrow  field-of- 
view  TV  would  be  used  to  identify  the  threat  as  friendly  or  foe.  Terrain 
following /te  r rain  avoidance  is  used  to  minimize  own-aircraft  exposure  and 
hence  reduce  the  susceptibility  of  the  strike  aircraft  to  detection  by  unfriendly 
forces. 

Ground  target  attack  sensors  include:  air-to-ground  radar  (con- 
ventional PP1  sector  scan,  high  resolution  DBS  and  SAR,  GMTI,  FTT  and 
AGR  modes),  TV,  FLIR  and  laser  spot  tracker.  Typical  operator  tasks 
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r ABLE  f>.  JT1DS  DISPLAY  SYMBOLOGY 


1 

FRIENDLY  AIRCRAFT 

2 

UNKNOWN  AIRCRAFT 

n 

A 

3 

HOSTILE  AIRCRAFT 

A 

/V 

4 

FRIENDLY  AIRBASE 

+ 

5 

HOSTILE  AIRBASE 

□ 

6 

HOSTILE  BASE 

7 

FRIENDLY  SAM 

8 

HOSTILE  SAM 

O 

9 

FRIENDLY  AAA 

: 

10 

HOSTILE  AAA 

11 

HOSTILE  GROUND  TRACK 

V 

12 

HOME  BASE 

U 

13 

OWN  SHIP 

O 

14 

DATA  POINT 

H- 

15 

WAY  POINT 

X 

16 

FEBA 



17. 

SAFE  AREA 

— 

18 

UNSAFE  AREA 

19. 

DOWNED  PILOT 

20 

CURSOR 

1 l 
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with  these  sensors  include:  parameter  and/or  mode  selection,  display 
brightness  and  contrast  adjustment,  terrain  orientation,  target  area  recog- 
nition, target  recognition,  target  identification,  target  designation  (sensor 
slewing  c.r  cursor  designation)  lock-on  command,  field  of  view  or  intra- 
sensor mode  change,  target  reacquisition  across  sensors  or  sensor  modes, 
and  commitment  for  weapon  delivery. 

Data  receiving  sensors  include  data  link  and  beacon.  These  sensors 
are  used  to  provide  the  operator  with  data,  typically  in  symbolic  form,  on 
friendly,  unknown,  and  hostile  aircraft;  SAM,  and  AAA  tracks;  own-base, 
data  point,  way  point,  FEBA,  and  target  locations;  safe  and  unsafe  areas; 
and  location  of  downed  pilots. 

Table  7 contains  a summary  of  the  sensor  information,  typical  dis- 
play format,  and  operator  tasks  for  baseline  and  desirable  F-18  sensors. 

3.4  SENSOR  UTILIZATION  AS  A FUNCTION  OF  MISSION  PHASE 

The  possible  uses  of  each  sensor  in  the  F-18  avionics  suite  were 
examined  for  various  phases  of  the  inte  rd  iction  mission  illustrated  in 
Figure  9.  Based  on  individual  sensor  capability  and  typical  operator  tasks 
associated  with  each  sensor,  Table  8 was  compiled  in  terms  of  potential 
sensor  utilization  per  mission  phase.  In  addition  to  F-18  baseline  and 
optional  sensors,  several  sensors  in  the  development  stage  were  included. 

Table  8 provides  a simplified  and  rapid  means  for  determining: 

1.  The  extent  to  which  any  individual  sensor  could  be  utilized 
throughout  an  entire  mission  (row), 

2.  The  multiplicity  of  sensors  which  might  be  utilized  during  a 
particular  phase  of  the  mission  (column)  and  could  thus  be  con- 
sidered for  simultaneous  presentation  on  a single  display. 

3.5  SENSOR  UTILIZATION  AS  A FUNCTION  OF  TARGET  TYPE 

Subsequent  to  determining  which  sensors  could  be  used  during  each 
phase  of  the  interdiction  mission,  the  sensors  were  further  subdivided 
according  to  targets  of  interest  as  summarized  in  Table  9.  Both  pre-briefed 
targets  and  targets  of  opportunity  were  considered  for  the  interdiction 
mission.  These  included  fixed  (or  mobile)  targets,  moving  targets  on 
ground,  sea,  or  in  the  air,  and  surface  and  air  emitters.  (See  glossary 
for  ac  ronyms.  ) 
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INFORMATION  AND  OPERATOR  TASKS 


TYPICAL  DISPLAY 


SENSOR 


OPERATOR  TASKS 


OPERATOR  SEARCHES,  ORIENTS 
HIMSELF  TO  GROUND  SCENE, 
DETECTS  AND  RECOGNIZES 
GROUND  OBJECTS,  DESIGNATES 
TARGETS 


SAME  AS  TV,  PLUS  THERMAL 
"HOT  SPOT"'  DETECTION 


AZ  VS  EL  PRESENTATION  OF  FLIR 
VIDEO,  THERMAL  "HOT  SPOTS" 


HIGH 

RESOLUTION 

TV 


AIR-TO-AIR.  MONITOR  TV  VIDEO 
TO  DETECT  AIRCRAFT.  LOCK-ON 
TO  DETECTED  AIRCRAFT. 
MONITOR  VIDEO  TO  IDENTIFY 
AIRCRAFT 


AIR  TO-GROUND  MONITOR 
VIDEO  TO  DETECT  AND 
IDENTIFY  GROUND  TARGETS 


TV  VIDEO  CUED  BY  HADAR  OR  VISUAL  DETECTION 


I 
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(Continued  next  page) 


(Tahir  7,  continued) 


AR 

NING 

-IVER 


MONITOR  RADAR  WARNING 
RECEIVER  DISPLAY,  DETECT 
AND  INTERPRET  THREAT 
WARNING,  INITIATE  APPRO- 
PRIATE COUNTERMEASURES 
(CHAFF,  FLARES,  VISUAL 
SEARCH.  EVASIVE  MANEUVER 
ETC) 


BEARING,  RANGE  AND  IDENTIFICATION 
OF  DETECTED  THREATS 


120°  AZIMUTH 
SCAN 


USED  FOR  TERRAIN  ORIENTA 
TION,  NAVIGATION  UPDATE, 
LARGE  TARGET  STRIKE 
OPERATOR  MONITORS  DISPLAY 
FOR  KNOWN  PREBRIEFED 
OBJECTS.  DESIGNATES  RADAR 
RETURNS 


DESIGNATING 

CURSOR 


AZ  VS  RANGE  RADAR  VIDEO 


DOPPLER 

BEAM 

SHARPENS' 

SECTOR 

PATCH 


AZIMUTH 


SAME  AS  REAL  BEAM  GROUND 
MAP  EXCEPT  IMPROVED 
AZIMUTH  RESOLUTION 
PERMITS  DETECTION  AND 
DESIGNATION  OF  SMALLER 
TARGET  OBJECTS  AND 
IMPROVED  DETECTION  AND 
DESIGNATION  OF  LARGE 
TARGETS 


HIGH  RESOLUTION  AZ  VS  RANGE  RADAR  VIDEO 
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(Continued  next  page) 


TA  380SB 


(Tabic 


continued) 


SYNTHETIC 
APERTURE 
RADAR 
STRIP  MAP 


SAME  AS  REAL  BEAM  GROUND 
MAP.  EXECPT  IMPROVED 
AZIMUTH  AND  RANGE  RESO- 
LUTION  PERMITS  DETECTION 
AND  DESIGNATION  OF  TAC- 
TICAL  TYPE  TARGETS  AND 
IMPROVED  DETECTION  AND 
DESIGNATION  OF  LARGE 
AND  MEDIUM  SIZED  TARGETS 


AZ  VS  RANGE  RADAR  VIDEO 


GROUND 

MOVING 

TARGET 

INDICATION 


MONITOR  RADAR  DISPLAY  FOR 
RADAR  DETECTED  MOVING 
TARGETS;  INTERPRET  MOVING 
TARGET  INDICATION  FOR 
STRIKE  SIGNIFICANCE.  DESIG- 
NATE GMTI  RETURN,  ACTIVATE 
AUTO  TRACK 


AZ  VS  RANGE  PRESENTATION  OF  DETECTED 
MOVING  GROUND  TARGETS  INTERLEAVED 
WITH  REAL  BEAM  GROUND  MAP 


LASER 

DESIGNATOR/ 

RANGER/ 

SPOT  TRACKER 


SYMBOL  ON  FLIR  AZ  VS  EL  DISPLAY  OR  RADAR  AZ  VS 
RANGE  DISPLAY,  ALSO  LASER  CODE,  LOCK-ON  INDICATION, 
AND  RANGE  TO  TARGET 


SET  UP  LASER  SYSTEM,  ENTER 
LASER  CODE,  DESIGNATE 
TARGET  FOR  LASING  OR 
RANGING.  OR  MONITOR  LASER 
SPOT  TRACKING  INDICATION, 
ACTIVATE  SYSTEM  MODES 


(Continued  next  page) 
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TA  3805C 


GROUND 
MOVING  OR 
FIXED  TARGET 
TRACK 


TRACKED  TARGET 
SYMBOL 

N 

B SWEEP^  ( 


MONITOR  AUTO  TARGET  TRACK, 
SEARCH  DISPLAY  FOR  OTHER 
TARGETS 


TRACKED  TARGET  SYMBOL  LOCATED  AT 
TARGET  RANGE  AND  AZIMUTH 


-120  AZIMUTH 
SCAN 


DETECT  BEACON  SYMBOL, 
CONFIRM  BEACON  CODE,  DE- 
SIGNATE BEACON  SYMBOL 


DESIGNATING 

CURSOR 


BEACON  SYMBOL  AT  AZIMUTH/RANGE  LOCATION 
OVERLAID  ON  REAL  BEAM  GROUND  MAP  VIDEO 


TERRAIN 

FOLLOWING 


MANEUVER 

TEMPLATE 


TERRAIN 

PROFILE 


TOWER 

SYMBOL 


LINEAR 

ELEVATION 

SCALE 


LOGARITHMIC 
RANGE  SCALE 


MONITOR  TERRAIN  PROFILE 
AND  MANEUVER  TEMPLATE; 
LOOK  FOR  TERRAIN  POINTS 
AND  TOWER  SYMBOLS  ABOVE 
MANEUVER  TAMPLATE  MA- 
NEUVER AIRCRAFT  TO  MAIN 
TAIN  DESIRED  CLEARANCE 
ABOVE  TERRAIN 


TERRAIN  ELEVATION  VS  RANGE  PROFILE,  COMPUTED 
MANEUVER  TEMPLATE,  TOWER  SYMBOLS 


(Continued  next  page) 


(Table  7,  continued) 


i 

I 


i 


TERRAIN 

AVOIDANCE 


DISPLAYED 
VIDEO  ABOVE 
SELECTED 
CLEARANCE 
PLANE 


MONITOR  DISPLAY  FOR  TER- 
RAIN ABOVE  CLEARANCE  PLANE 
ASSESS  AIRCRAFT  FLIGHT  PATH 
RELATIVE  TO  TERRAIN  OBSTA- 
CLES, MANEUVER  AIRCRAFT  TO 
AVOID  TERRAIN  OBSTACLES 


AZ  VS  RANGE,  RANGE  LINES,  VIDEO  OF  TERRAIN  ABOVE 
SELECTED  CLEARANCE  PLANE,  HORIZON  LINE  SYMBOL 


AIR-TO-AIR 

RADAR 


RANGE-WHILE- 
SEARCH  MODE 


ATTITUDE  AND 


MONITOR  RADAR  DISPLAY, 
DETECT  RADAR  HITS  ON 
DISPLAY,  INTERPRET  RADAR 
HITS  REGARDING  THREAT 
LEVEL,  DESIGNATE  RADAR 
HITS  FOR  LOCK-ON,  MONITOR 
LOCK-ON,  MONITOR  TRACK 
FILES,  COMMIT  MISSILES  FOR 
LAUNCH,  PERFORM  EVASIVE 
MANEUVERS 


AZ,  RANGE,  ELEVATION  BAR  OF  RADAR  HITS 
NO  OF  HITS,  TARGET(S)  FLIGHT  PATH 


AIR-TO-AIR 

RADAR 

SINGLE 
TARGET 
TRACK  MODE 


MONITOR  RADAR  DISPLAY, 
DETECT  RADAR  HITS  ON 
DISPLAY,  INTERPRET  RADAR 
HITS  REGARDING  THREAT, 
DESIGNATE  SELECTED  HIT, 
ACTIVATE  TRACK  MODE, 
MONITOR  RADAR  TARGET 
TRACK,  MONITOR  FIRE 
CONTROL  SOLUTION,  FIRE 
WEAPON 


AZ,  RANGE,  ELEVATION  BAR  OF  TRACKED  TARGET 
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(Continued  next  page) 


TA  3805E 


(Table  7,  concluded) 


DATA  LINK 


Al  PHANUMERIC/SYMBOLIC  CODED  DATA  OR  UNKNOWN, 
FRIENDLY,  AND  ENEMY  TRACKS.  AND  OTHER  LOCATIONS 
(e  g , HOME  BASE,  FEBA,  SAFE  AREAS,  DOWNED  PILOT,  ETC) 


MONITOR  DISPLAYED  DATA 
LINK  INFORMATION,  INTER- 
PRET DISPLAYED  INFORMA- 
TION 


IR  SEARCH 
AND 

WARNING 


/ V MONITOR  IR  WARNING  RECEI- 

/ N.  VER  DISPLAY  (CAN  HAVE 

/ N.  WARNING  AUDIO  OR  LIGHT 

/(  \ N.  PLUS  A/N  READOUT),  DETECT 

| AND  INTERPRET  THREAT 

m WARNING,  INITIATE 

■ APPROPRIATE  COUNTER- 

MEASURES 

■ 

l J 


TA  3805F 


1 - 2 

TAKE  OFF  AND  CLIMB 

2-3 

CRUISE 

3 

PRELIMINARY  TARGET  ACQUISITION 

4-5 

RUN-IN  TO  TARGET 

5-6 

POPUP 

6-7 

TARGET  ACQUISITION  AND  WEAPON  RELEASE 

7-8 

POSSIBLE  ASSISTANCE  IN  MISSILE  GUIDANCE 

8-9 

POSSIBLE  DAMAGE  ASSESSMENT 

9-10 

RETURN  TO  BASE 

Figure  9. 

Interdiction  mission  phases. 

Key  portions  of  the  mission  involving  targeting  were  re-examined 
including: 

1.  Acquisition  of  pre-briefed  tar  get /threat  data  before  take-off, 
and  updating  of  tar  get  /threat  information  while  in  flight  via 
data  link 

2.  Preliminary  target  acquisition  at  stand-off  range,  before 
descent  and  ingress 

3.  Target  acquisition  after  pop-up 

4.  Target  identification  and  weapon  release 

5.  Damage  assessment. 

Those  sensors  which  could  be  utilized  for  the  target  acquisition 
and  end  game  phases  of  the  interdiction  mission  were  obtained  from  the 
appropriate  columns  of  Table  8 and  listed  in  Table  9 as  a function  of 
mission  phase  and  target  type.  The  sensors  include  fixed  target  indication 
(FTI)  radar  such  as  RBGM,  DBS  and  SAR  mapping  and  track  modes,  and 
moving  target  indication  (MTI)  radar  for  both  air-to-ground  and  air-to-air 
target  detection  and  track.  Over-the-horizon  (OTH)  information  from 
satellite  radars  would  be  data-linked  to  the  aircraft. 


45 


07344  3719 


TABLE  8.  SENSOR  UTILIZATION  PER  MISSION  PHASE  - INTERDICTION 
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TABLE  9.  SENSOR  TECHNOLOGIES  FOR  TARGET  TYPES  AND 

MISSION  PHASES 


FIXED  TARGETS  (OR  MOBILEI 


MOVING  TARGETS 
GROUND  I SEA  I AIR 


EMITTERS 
SURFACE  I AIR 


PREBRIEFED  OR 
DATA-LINKED 
TARGET/THREAT 
INFORMATION 


■ CARTOGRAPHY  IGRIDI  — ► 

-FTI  RADAR  «. 

-IR  IMAGERY »- 

-PHOTO  (PTSI ► 


MTI RADAR 


-*-UGS 

GPS 


- JTIDS  ■ 


-OTH  RADAR  ■ 


■USS  ■ 


4 TEREC 

4 PELSS  - 

— ELS  

— TASES  - 


PRELIMINARY 
TARGET 
ACQUISITION 
(BEFORE  RUN-IN 
TO  TARGETI 


•FTI  RADAR 


H*- 

• BEACON - 


• MTI  RADAR  • 


• L/RWR  - 


FUR  HOT  SPOTS  ■ 

■ TV  BRIGHT  SPOTS  ►) 

■LASER  SPOT  TRACKER  — 


•♦-IRSS  • 


-MULTI-SOURCE  FUSION  ■ 


TARGET 
ACQUISITION 
(AFTER 
POP-UP  I 


•FTI  RADAR- 


-*+«- 


— MTI  RADAR  — ► 
-MICROWAVE  RADIOMETRY ► 


• BEACON 


■ LASER  DESIGNATOR/SPOT  TRACKER 
FLIR  • 


-TV 


♦-IRSS- 


• L/RWR  - 
FOLLOWED  BY 
RF  OR  EO 
PRECISION 
TRACKER 


•SIGNATURE  

MULTI-SOURCE  FUSION 


TARGET 

IDENTIFICATION 
AND  WEAPON 
DELIVERY 


•FLIR 
-TV  — 


• FTI/SPOTLIGHT  - 


•4-  MTI/SPOTLIGHT- 


4-NCTR  -► 


LASER  DESIGNATOR/RANGER/SPOT  TRACKER *• 

SPECIAL  SIGNATURE  


L/RWR 
AND  SPECIAL 
PROCESSING 


-AUTOMATIC  CUE/IDENT  SYSTEMS- 
-MULTI-SOURCE  FUSION  


DAMAGE 

ASSESSMENT 


•FLIR 
• TV  — 


■FTI  SPOTLIGHT 


-MTI  RADAR  

|4—  OTH  RADAR  *• 

• SIGNATURE  1 

MULTI-SOURCE  FUSION  


-FTI- 


SPOTLIGHT 
4 L/RWR  - 


MTI- 


F IX  ED  TARGETS 


GROUND  | SEA  | AIR 
MOVING  TARGETS 


SURFACE 


AIR 


EMITTERS 
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Two  factors  arc  evident  in  Table  9.  The  first  is  the  multi  spectral 
signatures  which  typically  characterize  a target,  whether  based  on  reflec- 
tivity or  emissions  (intentional  or  unintentional),  such  that  a target  is 
detectable  in  a number  of  spectral  regions  and  thus  by  a number  of  different 
sensors.  The  second  is  the  logical  grouping  of  those  sensors  according  to 
similar  operational  ranees.  In  the  following  section,  a sequence  of  multi- 
sensor simultaneous  combinations  based  on  these  factors  are  formulated 
for  possible  utilization  in  the  interdiction  mission. 

3.n  CONCLUSIONS 

As  a result  of  the  interdiction  mission  analysis  and  the  consideration 
of  a multiplicity  of  sensors  and  data  sources  normally  required  for  each 
phase  of  the  mission,  several  key  factors  emerge: 

1.  Different  mission  phases  have  different  information  requirements, 
and,  therefore,  different  real-time  sensor  and  a priori  data 
requirements. 

2.  The  use  of  prebriefed  a priori  information  stored  in  the  data 
base  and  retrieved  for  display  during  various  portions  of  the 
mission  can  be  extremely  important  in  reducing  operator  task 
load  and  increasing  the  probability  of  mission  success. 

3.  During  the  target  acquisition  phases,  the  multi-attribute 
characteristics  of  targets  permit  target  acquisition  within  a 
number  of  different  spectral  regions  and  thus  by  a number  of 
different  sensors.  The  simultaneous  use  of  multiple  sensors 
to  acquire  a particular  target  can  increase  the  probability  of 
successful  target  acquisition  and  may  decrease  the  search  time. 
Simultaneous  multisensor  capability  may  be  especially  valuable 

if  one  of  the  target  signatures  ceases  (such  as  an  emitter  shutting 
off  or  a moving  vehicle  stopping). 

4.  The  sensors  which  could  be  utilized  simultaneously  during  the 
various  mission  phases  fall  into  logical  groups  based  on  their 
operational  range  capabilities.  These  sensors  may  have  dif- 
ferences in  format,  number  of  scan  lines,  field  of  view,  reso- 
lution, and  update  rate,  thus  requiring  that  they  be  made 
compatible  before  they  can  be  presented  simultaneously. 

5.  Examination  of  the  tables  involving  sensor  utilization  as  a func- 
tion of  mission  phase  indicates  that  there  is  no  one  sensor  which 
provides  sufficient  information  to  act  as  the  lead  sensor  through- 
out the  entire  mission. 

The  following  section  attempts  to  1)  identify  combinations  of  critical 
or  primary  s ensors,  augmenting  sensors,  and  collateral  data  needed  for  key 
mission  phases;  and  2)  estimate  the  potential  utility  of  these  combinations  to 
operator  performance  and  mission  success. 
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4.0  CANDIDATE  SIMULTANEOUS  SENSOR  PRESENTATIONS 


In  this  section,  several  promising  simultaneous  presentations  are 
selected  which  warrant  future  investigation,  including  combinations  of  a 
real-time  sensor  with  other  sensor  data  and  with  stored  or  linked  collateral 
data.  These  selections  are  based  on  1 ) the  identification  of  a critical  or 
primary  sensor  associated  with  each  mission  phase,  and  2)  the  relative 
desirability  of  additional  sensors  and  collateral  data  for  each  mission  phase, 
as  determined  by  estimated  utility  of  that  information  to  the  operator  and 
estimated  criticality  of  that  information  to  the  mission  phase. 

In  general,  the  results  indicate  that  information  sources  which  could 
be  syner gi  stically  combined  are  range-to-target  dependent.  At  longer 
ranges,  these  include  radar  ground  map  as  the  primary  sensor  and,  as 
secondary  sources,  radar  beacon  and  air-to-ground  ranging  modes,  pre- 
briefed waypoint  data,  stored  maps  and  reconnaissance  imagery,  a priori 
and  linked  target/threat  data,  and  onboard  warning  receivers.  At  inter- 
mediate ranges,  promising  candidates  include  radar  (primary  sensor),  F LIR , 
GMTI,  A-G  ranging,  laser  ranger/ spot  tracker , warning  receivers,  and  a 
priori  and  linked  target/ threat  data.  At  closer  ranges,  EO  sensors  such  as 
FLIR  (primary  sensor)  and  TV  predominate,  augmented  by  the  use  of  warning 
receivers,  laser  spot  trackers/rangers,  and  possibly  some  types  of  radar 
data  and  automatic  target  cuing  information. 

In  an  attempt  to  evaluate  various  methods  of  combining  the  selected 
sensors  and  collateral  data  on  a single  display,  a data  base  of  available 
imagery  was  compiled.  Representative  static  imagery  was  then  super- 
imposed by  photographic  or  computer-aided  CRT  techniques  to  simulate 
several  of  the  selected  combinations.  The  resulting  photographs  were 
subjectively  evaluated  in  terms  of  the  relative  desirability  of  1)  the  primary 
sensor  display  only,  2)  the  primary  sensor  display  adjacent  to  the  secondary 
sensor  or  collateral  data  display,  and  3)  the  secondary  sensor  or  collateral 
data  (or  highlights  of  the  same)  overlaid  on  the  primary  sensor  display. 

This  section  describes  1)  the  selection  of  primary  and  secondary 
information  for  display  during  key  mission  phases,  2)  the  compilation  of 
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the  imagery  data  liase,  and  l)  the  potential  use  ol  the  simultaneous 
presentations  to  the  pilot  as  a function  of  mission  phase  (accompanied  hy 
simulated  multisensor  imagery  synthesi/.ed  from  the  clata  base  where 
available ) . 

Subsequent  sections  examine  these  selec  ted  simultaneous  presenta- 
tions in  terms  of  proi  essing  c on  s ide  rat  ion  s and  relative  c omplexity  of 
implementation. 

4.  1 SELECTION  OF  SIMl'I.I  ANKOUS  SENSOR  CANDIDA!  ES 

As  concluded  in  Section  i.  fl,  there  appears  to  be  no  one  sensor 
which  provides  sufficient  information  to  act  as  the  principal  or  lead  sensor 
throughout  the  entire  interdic  tion  mission.  Instead,  each  mission  phase* 
typic  ally  has  a different  primary  sensor  or  sensor  mode,  and  requires  one 
or  more  secondary  sensors  or  sets  of  supplementary  information  lor  suc- 
cessful task  completion.  Supplementary  information  includes  a priori  data 
and  imagery  stored  in  the  aircraft  data  base*  before  take-off  and  target/threat 
data  linked  to  the*  aircraft  in  flight. 

These  primary  sensors  and  secondary  sensors  or  supplementary 
(collateral)  information  are  identified  in  Table  10  as  a func  tion  of  key  mis- 
sion phases  including  cruise  (navigation),  preliminary  target  acquisition, 
run-in  to  target,  and  target  acquisition/weapon  delivery  at  pop-up.  The 
relative  utility  to  the*  operator  of  each  of  the  secondary  sources  of  informa- 
tion is  estimated,  as  well  as  I he  relative  criticality  of  that  information  to 
successful  completion  of  the*  particular  mission  phase. 

Based  on  these  ratings,  a number  of  combinations  emerge  which  are 
potentially  very  advantageous.  These  recommended  combinations  are  listed 
in  Table  10  in  terms  of  primary  sensor  imagery,  secondary  sensor  imagery 
(or  highlight  data)  and  secondary  collateral  imagery,  and  a number  of 
secondary  sensors  or  data  sources  whose  output  can  he*  displayed  as  symbology 
overlaid  on  the  primary  sensor  video. 

Note  that  within  any  single  mission  phase,  there  are  significant  dif- 
ferences among  the  various  types  of  necessary  or  desired  sensor /collateral 
information,  i.  e.  , one  sensor  or  set  of  data  does  not  necessarily  reinforce 
another  sensor's  output.  Note  also  the  gaps  which  can  exist  when  transi- 
tioning from  the  primary  sensor  of  one  mission  phase  to  the  primary  sensor 
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TABLE  10.  SIMULTANEOUS  PRIMAR  Y/SECONDAR  Y INFORMATION  - 
RELATIVE  UTILITY  AND  CRITICALITY 


Primary  Srninr 


Radar  i R BOM.  DBS, 
SARI 


Pi  eliminary 
Targrl 
A«  qui  ilium- 
I-ong  H »ngr 


Radar  I DBS,  SAR  i 


Sr.  ondary  Sensor  'Data  S.un  r 


I' » ...  on.  A-f  i R anging 

I |icr  R anger 


-Si - .red  map* 
si  . i rd  v»ay|Kiinl  imagery 
A priori  waypoint  into. 
Daia-ln  o-  i update  .i  a prion 
targ«  i Threat  info 
W at  rung  R.  t civet  a 

CiMTI 

A - G Ranging 
Laser  Ranger 
t UR.  T\ 

Stored  maps 

St,, red  n ager\  f target  are.. 
A >i  , .i  r get  threat  info 
I I.. la- linked  update  ■•!  a nf  i 


* stimated  1 tillty  to  P 1 t 
* erded  lor  long-range  na  igati-n 

Needed  ! r nav  igation  update 

Possible  | .t  nav  igatinn  update,  but  range 
limited 

possible  1 r na  igati  .n  update.  but  range 
limited 

Needed  for  nav  i gat  ion 

Helpful,  but  typually  not  available 

\e.  cssary 

aluablc.  P.  prepare  for  target 
a.  qui  situui  defensive  a.  tion 
\e«  essary 

dell  for  long-range  target  acquisition 
I . Iplut,  but  range  and  aiiuiai)  limited 
Range  limited 
It  .mgr  lini  i »-d 

Rangt  Lnoi.  d d.  u«  nding  «n  target) 

I le  I piu  I 

\ aluahle.  it  available 

V ..luable 

V aluabl. 


I stimated 
C r itii  ality 


M derate 
" n.gii 

Mode  r ate 
N1-  - ■ I • rate 


Moderate 

High 

H.gh 

High 


mg 

Helpful.  liut 

•g*  r 

Helpful,  but 

t Trac  ker 

1 ’ . amble 

r (Hint  a 

llelptul  t..r  dete.  tion  of  ho 

or  i old  target 

a 1 liable  t..r  m • .ng  large 

i ii. ng 

It  anging 

\ aluahle  for  weapon  deltve 

ry  . omputat  ion 

r R angr r 

Valuable  for  weapon  delive 

ry  . .imputation 

r Spot  T r a.  Irr 
■d  mapa  / Image  r y 

Valuable  for  AFAC-design 
V aluable  for  target  detail 
1 I.IR  if  av  a 1 1 able 
Unlikely 
Valuable 

ated  target* 
s asst  st  to 
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of  the  next;  for  example,  the  difference  in  scene  characteristics  when 
switching  from  radar  imagery  (preliminary  target  acqui  sition)  to  FUR 
imagery  (target  acquisition  at  pop-up). 

The  simultaneous  display  of  the  information  listed  in  Table  1 1 repre- 
sents one  potentially  valuable  method  of  gathering  together  or  integrating  the 
essential  elements  of  information  on  a single  monitor  (or  common  grid,  so 
to  speak)  and  smoothing  the  transition  from  one  primary  sensor  to  the  next. 

It  combined  and  sequenced  appropriately  throughout  the  mission,  these 
simultaneous  presentations  oiler  significant  promise  of  decreasing  the  opera- 
tor task  loading  throughout  the  mission  while  increasing  survivability  and 
the  probability  of  successful  target  acquisition  and  weapon  delivery. 

Each  of  the  recommended  combinations  are  described  in  more  detail 
in  this  section,  accompanied  by  synthesi/.ed  imagery  where  available. 

TABLE  11.  RECOMMENDED  MULTISENSOR  / DATA  COMBINATIONS 


Mission  Phase 

Primary  Sensor 

Su  ondary  Sensor 
Collateral  Data 

Supplementary  S.  nsors/ 
Data  Symbology 

Cruise/Navigatmn 

Radar  Imagery 

• R BG  M 

• DBS 

• SAR 

Map 

• Beacon,  A-G  Ranging 

• P rebriefed  CP's 

• 

• Stored/ Linked 

Threat  Data 



Preliminary  Target  Acquisition 
• Long  H ang*- 

Radar  Imagery 

• DBS 

• kAR 

Stored  Imagery  t Target  Area 

Symbology 

• Stored/  Linked 
Target  Data 

• GMT! 

• timing  Re  eivei 

• Stored  I inked 
Threat  Data 

• Intermediate  Range 

Radar  Imagery 

• DBS 

• SAR 

FLIR 

• Hot  Spots 

Symbology 

• ctractiot 
Pattern  Re.  .ignition 
Devu  e s 

• Lasei  ickcr 

Range  r 

• GMT1.  A-G  Ranging 

• Warning  Receiver  a 

• Stored  Linked 
Target  Threat  Data 

Target  Acquisition  at  Pop-L'p 

• Stand-Off  Range 

• Stand-Off  to  Minimum 
Range 

FLIR  Imagery 

FLIR  Imagery 
• Prot  eased 

R adar 

• Glints 
TV  Imagery 

• Processed 

Symbology 

• Automat  i.  Target 
Cuing  Devices 

• Laser  Spot  Tracker 
Designator ' Ranger 

• w d rmng  Receivers 

• Stored / I. ink ed 
Target /Threat  Data 

• GMT1.  GMTT 
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4.  2 ACQUISITION  OF  DATA  BASE 

In  order  to  qualitatively  evaluate  candidate  multisensor  combinations, 
static  samples  of  sensor  imagery  were  used  to  simulate  the  multisensor 
displays.  To  acquire  the  imagery  needed  for  synthesis,  a comprehensive 
inquiry  of  military  sources  was  made  to  locate  existing  coincident  coverage 
multisensor  imagery  of  tactical  and  strategic  targets.  Due  to  the  paucity  of 
applicable  static  and  dynamic  imagery,  a flight  test  was  conducted  in  which  a 
FLIR  and  a TV  camera  were  mounted  on  a Hughes  aircraft  and  flown  over  a 
flight  path  corresponding  to  previously  obtained  radar  imagery.  From  the 
r.sulting  data  base  of  static  and  dynamic  sensor  imagery,  a representative 
sample  of  radar,  aerial  photography,  F LIR  and  TV  imagery  was  selected 
for  static  simulation  of  the  various  combinations  either  by  projection  through 
transparencies  or  by  display  on  a CRT.  The  collection  of  dynamic  and  static 
imagery  for  the  data  base  is  described  in  the  following  paragraphs. 

4.  2.  1 Dynamic  Imagery 

To  obtain  dynamic,  real-time,  simultaneous  coincident-coverage 
imagery  from  several  sensors  for  superposition  and  visual  evaluation,  a 
flight  test  was  conducted  using  the  Hughes  Convair  aircraft.  Sensors 
onboard  included  the  Hughes  HIPOD  FLIR  (315-line  serial  scan)  and  a con- 
ventional daytime  TV  with  similar  but  not  equal  fields  of  view.  While  the 
FLIR  was  slewable  over  the  lower  hemisphere,  the  installation  of  the  TV 
was  limited  to  look  angles  between  nadir  and  45  degrees,  along  the  flight 
path.  Therefore,  both  sensors  were  set  at  45  degrees  to  most  closely 
approximate  the  forward-looking  angles  associated  with  target  acquisition 
and  weapon  delivery.  Accurate  boresighting  was  not  attempted.  A flight 
plan  was  selected  to  be  coincident  with  existing  high  resolution  radar  imagery 
obtained  with  the  Hughes  Forward-Looking  Advanced  Multimode  Radar 
(FLAMR).  Target  areas  included  several  local  harbors,  airports  and 
refineries.  Thus,  imagery  of  the  same  scene  was  obtained  for  three  sen- 
sors — FLIR,  TV  and  synthetic  aperture  radar,  although  the  SAR  data  did 
not  coincide  in  time.  The  FLIR  imagery  was  recorded  on  both  16  mm  film 
and  video  tape  while  the  TV  was  also  recorded  on  video  tape.  Audio  descrip- 
tions of  flight  parameters  and  scene  characteristics  recorded  on  both  FLIR 
and  TV  tapes  assisted  in  subsequent  synchronization  of  the  two  images  when 
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displayed  side-by-side.  The  data  were  viewed  simultaneously  and  the 
differences  in  image  characteristics  noted  for  assistance  in  determining 
what  information  from  one  sensor  might  be  valuable  to  superimpose  on  the 
second  sensor  display.  Video  superposition  was  not  attempted. 

The  second  collection  of  simultaneous  sensor  video  involved  ground- 
based  measurements  of  ground  and  maritime  targets.  It  utilized  a low 
resolution  F LIR  and  TV  which  had  matched  fields  of  view  and  were 
boresighted.  Manual  correction  of  the  boresight  alignment  was  provided. 

The  two  simultaneous  sensor  video  were  automatically  summed  and  dis- 
played on  a display  monitor.  The  video  gains  were  manually  controlled  such 
that  FLIR  only  could  be  displayed,  TV  only,  or  any  intermediate  weighted 
sum  ( such  as  0.  75  FLIR  plus  0.  25  TV,  0.  5 FLIR  plus  0.  5 TV,  or  0.  25  FLIR 
plus  0.  75  TV,  etc.  ).  As  the  superimposed  video  was  viewed  on  the  monitor 
and  adjustments  in  relative  gain  and  boresight  alignment  were  made,  the 
data  were  video-taped.  Still  photographs  of  various  targets  and  video  gains 
were  taken  for  subsequent  side-by-side  comparison  and  qualitative  evalua- 
tion in  this  program. 

4.  2.  2 Static  Imagery 

Static  imagery  of  tactical  targets  obtained  by  various  sensors  onboard 
various  aircraft  were  collected  in  the  form  of  photographs,  negatives,  and 
transparencies.  Although  most  of  the  imagery  was  not  acquired  by  sensors 
operating  simultaneously  onboard  a single  aircraft,  it  was  possible  to  obtain 
imagery  of  fixed  tactical  targets  taken  by  a number  of  different  sensors  at 
approximately  the  same  time  of  day  and  under  similar  weather  conditions. 
These  sensors  included  synthetic  aperture  radar,  down-looking  infrared 
linescanner,  aerial  reconnaissance  camera,  and  corresponding  cartography. 
The  data  can  be  grouped  into  the  following  sets. 

Down- Looking  Infrared  and  Reconnaissance  Camera  Hardcopy 

The  same  regions  are  represented,  but  the  imagery  is  not  coincident 
in  time  or  scale  factor.  Some  measure  of  superposition  was  possible  (to 
simulate  IR  plus  TV)  by  matching  the  magnification  ratios  using  photographic 
enlargement  techniques,  generating  matched  transparencies,  and  then 
projecting  light  through  the  two  superimposed  transparencies.  However, 
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portions  of  the  images  were  sufficiently  misregi  stered  that  the  results  were 
inconclusive.  Accurate  registration  would  have  required  that  the  two  images 
be  digitized  and  spatially  warped  using  a computer  and  sophisticated  warping 
algorithms.  The  scope  of  this  contract,  however,  precluded  the  development 
of  such  algorithms. 

Radar  and  Cartography 

Hardcopy  of  synthetic  aperture  radar  imagery  and  corresponding 
cartography  were  scaled  to  the  same  magnification  factor  using  photographic 
techniques.  Light  was  then  projected  through  the  two  scaled,  superimposed 
transparencies  and  the  resulting  image  photographed.  This  technique  was 
used  to  simulate  a radar  display  with  overlaid  map  data  (including  lines  of 
communication  such  as  roads,  rivers  and  railroad  tracks). 

Radar  and  Reconnaissance  Photographs 

Radar  imagery  and  aerial  photography  were  used  for  two  purposes: 

].  To  simulate  the  superimposed  display  of  real-time  radar  video 
and  stored  reconnaissance  photos  for  purposes  of  target  acquisi- 
tion or  verification,  and, 

2.  To  simulate  the  display  of  real-time  radar  video  and  super- 
imposed highlight  information  from  other  real-time  sensors  such 
as  FLIR  or  TV. 

In  the  first  case,  radar  image  transparencies  (in  black/white  or 
green)  and  transparencies  of  reconnaissance  photos  (black/white)  of  the 
same  scene  were  scaled  to  the  same  size  photographically.  The  resulting 
transparencies  were  superimposed  and  visually  registered  by  aligning  key 
features  common  to  both  images.  The  resulting  image  was  then  photographed 
in  black  and  white  and  in  color. 

In  the  second  case,  information  was  extracted  from  the  aerial  photo 
to  simulate  highlight  information  which  might  be  obtained  from  a FLIR  or 
TV  sensor.  In  general,  FLIR  or  TV  imagery  which  is  presented  in  azimuth/ 
elevation  must  undergo  a spatial  warping  to  be  made  compatible  with  a radar 
image  which  is  presented  in  azimuth/range.  Because  the  development  of 
these  warping  algorithms  was  beyond  the  scope  of  this  study,  the  available 
radar  imagery  and  coincident  coverage  FLIR  and  TV  imagery  could  not  be 
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adequately  superimposed.  Instead,  down-looking  reconnaissance  photos 
were  scaled  to  match  their  counterpart  synthetic  aperture  radar  images, 
which  also  appear  to  be  vertical  or  down-looking  due  to  radar  processing 
techniques.  The  radar  and  photo  transparencies  were  then  digitized  and 
appropriately  formatted  on  tape  for  entry  into  an  interactive  computer/ 
display  system.  Various  image  processing  techniques  were  used  to  extract 
enhanced  or  highlight  information  from  the  photo  to  represent  FLIR  hot  spot 
or  TV  bright  spot  information.  This  highlight  data  was  then  displayed 
simultaneously  with  the  radar  video  on  a CRT  monitor  in  black  and  white 
and  in  a number  of  different  color  combinations.  The  CRT  was  photographed 
in  color  and  the  results  used  to  qualitatively  compare  the  various  enhancing 
techniques  and  the  various  color  combinations. 

A large  number  of  simultaneous  sensor  simulations  were  synthesized 
from  the  data  base  that  has  been  described  here.  Several  samples  from 
these  simulations  have  been  selected  to  illustrate  and  assist  in  evaluating  the 
multisensor  display  candidates.  These  selections  are  presented  in  the  next 
sub  section. 

4.  3 DESCRIPTION  OF  SIMULTANEOUS  PRESENTATION  CANDIDATES 
ACCORDING  TO  MISSION  PHASE 

This  section  attempts  to  accomplish  two  tasks  simultaneously: 
firstly,  to  describe  the  manner  in  which  information  from  the  primary  sensor 
and  secondary  data  sources  associated  with  each  mission  phase  could  be 
combined  on  a single  display;  and,  secondly,  to  describe  the  way  in  which 
the  combined  information  might  assist  the  operator  in  performing  the  major 
tasks  associated  with  each  mission  phase. 

The  discussion  is  divided  into  three  types  of  information  combinations 
which  correspond  to  three  different  portions  of  the  mission.  These  include 
the  combination  of  1)  real-time  radar  and  stored  cartography  for  navigation, 

2)  real-time  radar  and  stored  (or  linked)  reconnaissance  imagery  for  long 
range  preliminary  target  acquisition,  and  3)  several  real-time  sensors 
operating  simultaneously  for  closer  range  target  acquisition  and  weapon 
delivery.  The  recommended  primary  sensors  and  secondary  data  sources 
for  the  various  phases  were  previously  identified  in  Table  11.  Where 
available,  synthesized  imagery  from  the  data  base  accumulated  during  the 
course  of  the  study  has  been  included  to  simulate  candidate  display 
combination  s. 
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4.3.1  Navigation  I sing  a Real-Time  Sensor  and  Stored  Cartography 

During  the  cruise  phase  of  the  interdiction  mission,  the  primary 
sensor  has  been  identified  as  the  radar.  The  essential  task  the  operator 
performs  with  the  air-to-ground  radar  throughout  cruise  is  navigation  update. 
This  can  be  accomplished  with  varying  degrees  of  accuracy  depending  on  the 
available  navigation  aids  and  on  the  performance  of  the  available  A-G  radar 
modes  — 1)  real  beam  ground  map,  2)  doppler  beam  sharpened  (DBS)  sector 
and  3)  DBS  patch,  or  4)  synthetic  aperture  radar  (SAR)  passing  scene  and 
5)  SAR  telescope  track.  Secondary  information  for  the  task  of  navigation  to 
prebriefed  target  areas  is  a map  with  waypoints  clearly  marked. 

One  possible  means  of  simplifying  the  navigation  update  task  is  the 
superposition  of  prebriefed  map  data  and  checkpoint  symbology  on  the  radar 
imagery'.  An  incrementally  "moving''  map  display  could  be  electronically 
generated  from  map  data  stored  on  tape  in  the  aircraft  data  base  and  overlaid 
on  the  real-time  radar  display.  The  map  overlay  could  be  monochrome, 
although  color  is  probably  more  effective.  In  addition  to  waypoints,  shape- 
ceded  or  color-coded  symbology  could  be  superimposed  on  the  radar /map 
combinations  to  represent  beacon  information,  prebriefed  or  data-linked 
threat  location  and  identification,  and  real-time  threat  data  obtained  from 
onboard  warning  receivers.  The  appropriate  choice  of  colors  could  be  used 
to  improve  discrimination  between  graphics  and  symbols. 

Superposition  of  map  data  on  radar  imagery  at  selected  points  in  the 
mission  would  eliminate  the  need  to  look  back  and  forth  from  the  radar 
display  to  the  displayed  or  hand-held  map  information.  Also,  the  use  of 
overlaid  cartography  might  allow  a lower  resolution,  less  expensive  radar 
to  be  utilized  than  would  ordinarily  be  required. 

4.  3.  2 Preliminary  Target  Acquisition  Using  a Real-Time  Sensor 

and  Stored  Imagery 

Preliminary  target  acquisition  of  prebriefed  fixed  targets  typically 
occurs  in  the  latter  stage  of  cruise  and  preferably  at  long  range.  The 
primary  sensor  for  long  range  target  acquisition  is  typically  the  radar,  using 
either  the  doppler  beam  sharpened  sector  or  patch,  or  a synthetic  aperture 
radar  mode  which,  if  available,  could  extend  the  detection  range  of  large, 
fixed,  prebriefed  targets  to  ISO  nm.  Supplementary  information  for  pre- 
briefed targets  typically  consists  of  reconnaissance  imagery  of  the  target 
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area  which  is  stored  in  the  aircraft  data  base  and  automatically  called  up  for 
display  at  the  appropriate  time  in  the  mission.  This  stored  imagery  obtained 
on  some  previous  flight  could  be  radar,  down-looking  or  forward-looking  IR, 
laser  line-scan,  aerial  photo,  etc.  , and  may  have  a dissimilar  viewing 
aspect  compared  to  the  real-time  sensor. 

Presently,  the  pilot  usually  holds  the  target  photo  on  his  lap  and 
visually  correlates  the  photo  with  distinctive  features  on  his  radar  display. 

A more  desirable  system  would  have  the  imagery  stored  on  tape  and  pre- 
sented on  a second  display,  preferably  side  by  side  if  panel  space  permitted, 
for  ease  of  comparison.  A third  technique  is  the  superposition  of  radar  and 
stored  target  imagery  on  a single  display,  which  could  serve  several  pur- 
poses ranging  from  increased  information  content  on  a single  display,  to 
target  verification,  to  more  accurate  offset  aimpoint  designation  for  weapon 
delivery  computations.  In  both  the  side-by-side  and  superimposed  cases, 
the  mission  computer  keeps  track  of  the  real-time  radar  image  coordinates 
and  prebriefed  target  coordinates  (by  which  the  stored  target  imagery  is  filed), 
so  that  the  stored  imagery  can  be  called  up  at  the  appropriate  time  for 
di  splay . 

As  an  example  of  preliminary  target  acquisition  using  real-time 
radar  ground  map  and  stored  target  area  imagery,  Figure  10a  represents  a 
low-resolution  (80  ft.  ) synthetic  aperture  radar  image  of  an  air  base  at  a 
range  of  about  25  nmi,  shown  here  in  black  and  white.  Figure  10b  is  a down- 
looking reconnaissance  photo  of  the  target  area,  also  shown  in  black  and  white. 
To  simulate  the  combined  display  of  these  two  images,  the  two  transparencies 
were  overlaid,  aligned,  and  the  resulting  image  was  photographed,  as  shown 
in  Figure  lOd.  To  evaluate  the  effect  of  color,  the  radar  image  was  also 
presented  in  green,  as  shown  in  Figure  10c,  and  the  combination  of  the  green 
radar  image  and  the  black  and  w'hite  photo  is  shown  in  Figure  lOe. 

Figures  10a  and  d or  c and  e can  be  used  to  estimate  the  possible 
utility  of  the  combined  presentation  of  radar  and  photo  as  compared  to  radar 
only.  In  general,  the  combined  presentation  provides  additional  information 
such  as  lines  of  communication  (roads,  railroad  tracks,  etc.)  which  were 
not  imaged  on  the  radar  display.  Also,  the  superposition  of  the  photo  can 
assist  in  offset  aimpoint  designation.  For  example,  if  the  specific  target 
had  been  marked  on  the  stored  photo  (say,  within  the  box  outlined  in  black), 
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a LOW  RESOLUTION  SAR 
(BLACK  WHITf 


b DOWN  LOOKING  PHOTO 
(BLACK  WHITE! 


c LOW  RESOLUTION  SAR  (GRFEN 


d SUMMATION  Of  SAR  PLUS 
PHOTO  (BLACK  WHITE) 


e SUMMATION  Of  SAR  I GRFFN) 
PLUS  PHOTO  (BLACK  WHITE! 


FIGURE  10  SIMULATED  SIMULTANEOUS  PRESENTATION  Of  LOW  RESOLUTION  SAR  IMAGE  AND  RECONNAISSANCE 
PHOTO 


the  designation  of  that  point  by  a movable  cursor,  after  superposition  of  the 
photo  on  the  radar,  would  enable  the  target  coordinate s /referenced  to  the 
radar)  to  be  entered  into  the  mission  computer.  During  descent  and  low- 
altitude  penetration,  the  computer  would  continue  to  update  coordinates 
relative  to  the  aircraft,  such  that  at  pop-up,  an  EO  imaging  sensor  could 
be  automatically  slewed  to  the  appropriate  line  of  sight. 

In  practice,  radar/photo  summation,  as  exemplified  by  Figure  lOd, 
may  present  too  much  data  to  the  pilot.  It  may  be  preferable  to  extract  only 
salient  information  from  the  stored  image  for  display  on  the  radar.  For 
example,  the  photo  could  be  thresholded  such  that  only  the  brightest  shades  • 
of  gray  would  be  superimposed  on  the  radar,  possibly  in  a different  color. 

Or  the  photo  could  be  edge-enhanced  such  that  only  "outlines'  would  be  over- 
laid on  the  radar,  thus  providing  major  lines  of  communication  as  well  as 
outlining  of  high  contrast  areas.  In  this  manner,  the  operator  can  be  pro- 
vided the  required  information  without  significantly  increasing  the  task  of 
interpretation. 

In  general,  the  use  of  false  color  coding  in  combined  displays  may 
assist  in  differentiating  information  or  imagery  from  several  sources.  The 
use  of  green  for  the  radar  image  in  the  combination  of  Figure  lOe  does  not 
appear  to  offer  significant  advantages.  However,  if  it  had  been  possible  to 
also  color  code  the  photo,  for  example,  in  red,  more  distinct  definition  of 
radar  glints  from  bright  objects  in  the  photo  would  have  been  possible. 

The  display  of  important  additional  data  can  be  achieved  by  over- 
laying symbology  on  the  combined  imagery  of  either  Figure  lOd  or  e.  On  the 
black/white  display,  the  symbology  can  be  intensity-coded,  shape-coded,  and 
even  flashed  to  increase  discrimination.  A color  display,  however,  has  the 
advantage  of  color-coded  symbol  capability.  In  either  case,  critical  infor- 
mation such  as  emitter  locations  and  identifications  obtained  from  warning 
receivers,  threat  data  stored  in  the  data  base  or  linked  to  the  aircraft, 
moving  target  indications,  and  a variety  of  other  data  can  be  provided  in  an 
attention-focusing  manner  within  the  context  of  the  primary  sensor  imagery. 

As  in  the  previous  case  of  combined  radar  and  cartography,  the  use 
of  imagery  (raw  or  processed)  superimposed  on  the  radar  display  may 
enable  a lower  resolution,  less  expensive  radar  to  be  utilized  than  would 
ordinarily  be  required. 
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4.  3.  3 Target  Acquisition  Using  Real-Time  Sensors 


Three  representative  target  acquisition  cases  are  considered: 

1.  Preliminary  target  acquisition  at  long  range,  descent,  run-in, 
and  pop-up  at  stand-off  ranges  greater  than  50,  000  feet. 

2.  Preliminary  target  acquisition  at  intermediate  range,  descent, 
run-in,  and  pop-up  at  stand-off  ranges  less  than  50,000  feet. 

3.  Minimum  range  pop-up. 

C ase  1 

The  primary  sensor  for  preliminary  target  acquisition  at  long  range 
is  ground  mapping  radar,  the  desirable  modes  being  doppler  beam  sharpened 
sector  or  ; ->tch,  and  synthetic  aperture  radar  (SAR)  passing  scene  or  tele- 
scope track.  Assume  preliminary  target  acquisition  in  this  case  was  accom- 
plished using  low  resolution  SAR  with  the  possible  superposition  of  prebriefed 
map  or  reconnaissance  imagery  retrieved  from  the  aircraft  data  base.  The 
aircraft  then  descends  for  low-altitude  penetration.  If  long  stand-off  range 
pop-up  (in  excess  of  50,  000  feet)  is  desired  by  the  pilot,  then  the  radar 
ground  map  typically  remains  the  primary  sensor  at  pop-up.  However,  it 
is  possible  that  the  high  performance  F L1R  may  also  be  capable  of  providing 
useful  information  at  these  ranges.  This  FLIR-derived  information  could 
be  superimposed  on  the  radar  display  to  augment  the  radar  information  or 
cue  the  pilot  by  indicating  "hot  spot"  locations,  or  large  target  complexes. 

To  simulate  this  displayed  combination  of  primary  radar  imagery 
and  FLIR  highlight  data  at  stand-off  range  pop-up,  the  SAR  image  and  optical 
image  previously  shown  in  Figure  10  were  digitized  and  stored  on  magnetic 
tape.  The  radar  and  simulated  F'LIR  data  were  then  displayed  on  a CRT  in 
black/white  and  various  colors  and  photographed.  Samples  of  the  results 
are  shown  in  Figure  11  for  purposes  of  comparison. 

Figure  11a  represents  the  radar  ground  map  at  pop-up,  displayed  in 
black  and  white.  Low  resolution,  poor  quality  imagery  was  deliberately 
chosen  here  to  simulate  the  type  of  radar  imagery  upon  which  the  pilot  must 
often  depend.  To  simulate  the  combination  of  radar  and  FLIR  highlight 
imagery  at  pop-up,  the  radar  image  of  Figure  11a  has  been  summed  pixel 
by  pixel  with  the  optical  imagery,  each  with  a gain  of  0.5.  This  simple 
mixing  of  the  two  images  in  black/white  is  shown  in  Figure  lib.  To  estimate 
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the  possible  value  ot  taise  color  coding  of  the  two  sensor  outputs,  Figure  lie 
shows  the  radar  in  green  and  the  simulated  FLIR  in  red.  In  Figure  lid, 
the  radar  remains  green  while  the  FLIR  information  is  in  blue.  In  Figure  lie, 
the  radar  ground  map  is  displayed  in  red  and  the  FLIR  in  green.  In  Fig- 
ure Ilf,  the  radar  is  in  blue,  the  FLIR  in  red.  At  closer  ranges,  during 
target  recognition  and  weapon  delivery,  the  electro-optical  sensor  (FLIR 
or  TV)  simulated  by  Figure  1 1 g replaces  the  radar  as  the  primary  sensor. 

F rom  the  simulated  multi  sensor  di  splay  s of  F igure  11,  color  coding 
of  the  two  sources  of  information  appears  preferable  to  the  black/white 
combination  for  this  particular  example.  Comparison  of  Figures  11c  and  e 
illustrate  why  red  should  be  used  primarily  for  cuing  rather  than  for  back- 
ground imagery.  The  red  radar  background  distracts  the  attention  from  the 
points  of  interest  such  as  the  green  air  strip  and  lines  of  communications. 

The  opposite  combination,  red  on  green,  appears  far  superior.  Figure  Ilf 
illustrates  the  3-dimensional  effect  which  is  obtained  when  blue  is  used  for 
the  background  imagery  and  supplementary  information  is  presented  in  red. 

(In  this  case,  the  optical  image  was  differentiated  to  produce  the  edge- 
enhanced  or  outlining  effect  shown.  ) 

The  simple  mixing  of  the  two  sensor  outputs  shown  in  Figures  lib 
through  Figure  lie  may  result  in  an  excess  of  data  and  may  also  mask 
points  showing  strong  radar  glints  or  FLIR  hot  spots.  An  alternate  technique 
is  to  process  the  FLIR  output  so  as  to  extract  salient  information,  then 
superimpose  only  that  processed  or  enhanced  FLIR  information  on  the 
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radar  display.  As  an  example,  Figure  12a  shows  the  simple  mixing  of  radar 
in  green  and  simulated  FLIR  in  blue,  each  at  a gain  of  0.  5.  By  comparison, 
in  Figure  12b,  an  edge  enhancement  technique  has  been  used.  The  radar  is 
again  displayed  in  green  while  a simple  linear  differentiation  has  been  applied 
to  the  simulated  FLIR  image  and  the  resulting  "outlines1'  are  displayed  in 
blue.  This  process  results  in  an  improved  definition  of  major  lines  of  com- 
munication as  well  as  outlining  of  high  contrast  objects. 

In  Figures  12c  and  d,  the  appropriate  radar  pixels  are  "replaced" 
with  the  FLIR  highlight  data  to  prevent  masking  of  FLIR  information.  In 
Figure  12c,  the  optical  or  FLIR  image  has  been  thresholded  at  90  percent 
such  that  only  features  or  objects  with  the  greatest  temperature  differentials 
would  be  placed  on  the  radar  display.  Since  each  blue  pixel,  representing  a 
thresholded  FLIR  element,  replaces  a green  radar  pixel,  some  strong  radar 
returns  may  be  lost.  To  avoid  this  loss  of  potentially  useful  radar  informa- 
tion, a third  color  could  be  used  to  denote  those  points  at  which  both  the  radar 
signal  and  the  FLIR  signal  were  above  a certain  level.  In  Figure  12d,  the 
FLIR  has  been  thresholded  at  70  percent  of  maximum  intensity,  the  ''brightest'' 
shades  of  gray  are  displayed  in  blue  in  place  of  the  radar,  and  those  points  at 
which  both  the  radar  and  the  FLIR  exceed  70  percent  of  their  respective 
maxima  are  color  coded  in  orange. 

Ultimately,  it  should  be  possible  with  sophisticated  image  processing 
techniques  to  perform  pattern  recognition  and  feature  extraction  operations 
on  one  sensor  image  and  display  that  information  by  means  of  graphics  and 
symbology  either  on  the  same  sensor  image  or  on  a second  sensor  image. 

LOC s such  as  roads  and  railroad  tracks  could  be  extracted  and  displayed  as 
maximum  intensity  or  color-coded  lines  on  the  radar  display.  Vehicles 
such  as  tanks  and  trucks  could  be  recognized  with  estimated  confidence  levels 
by  appropriate  processing  of  FLIR  or  radar  returns.  This  target  cuing 
information  could  then  be  displayed  as  a coded  symbol  at  the  proper  location 
on  the  FLIR  or  radar  display. 

On  each  of  the  synthesized  sensor  combinations  shown  in  Figures  11 
and  12.  color-coded  and  shape-coded  symbology  could  be  used  for  other 
sensor  data  such  as  CIMTI  and  warning  receivers,  as  well  as  stored  and 
linked  threat  data. 
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SUMMATION  OF  SAR  PLUS  FLIR 


FIGURE  12  SIMULTANEOUS  PRESENTATION  OF  SAR  IMAGERY  AND  PROCESSED  SIMULATED  FLIR  IMAGERY 


c SAR  PLUS  THRESHOLDED  (90"..)  FLIR 


d SAR  PLUS  THRESHOLDED  (70%l  FLIR  PLUS 

COINCIDENT  RADAR  GLINTS  AND  FLIR  HOT  SPOTS 
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C ase  2 


This  case  pertains  to  intermediate  stand-off  range  pop-up  which  is 
applicable  to  three  types  of  targets:  1)  prebriefed  targets,  2)  targets  handed 

off  via  laser  designation,  as  in  hunter-killer  teams  or  situations  involving 
an  AFAC,  and  3)  targets  of  opportunity  located  in  prebriefed  target-rich 
areas.  Two  examples  related  to  targets  of  opportunity  will  be  considered 
here:  firstly,  the  combined  use  of  radar  as  the  primary  sensor  and  FLIR  as 

the  augmenting  sensor  to  acquire  targets  of  opportunity;  and  secondly,  the 
use  of  FLIR  as  the  primary  sensor  and  radar  as  the  augmenting  sensor. 

These  examples  attempt  to  illustrate  how  FLIR  hot  spot  information 
can  be  used  to  cue  the  operator  to  potential  targets  on  the  radar  display, 
how  retained  radar  imagery  can  provide  a context  for  narrow  f ield-of- view 
FLIR  imagery,  and  how  radar  glint  information  can  be  used  to  cue  the 
operator  to  targets  on  the  FLIR  display.  The  examples  are  described 
separately.  However,  in  practice,  they  can  be  sequenced  for  use  in  one 
target  acqui  sition/ weapon  delivery  operation. 

Radar  and  FLIR 

The  primary  sensor  for  preliminary  target  acquisition  is  assumed 
to  be  the  radar.  The  example  shown  in  Figure  13a  is  high  resolution  (10  ft.  ) 
SAR  at  a range  of  about  15  nmi.  displayed  in  black  and  white.  A river 
crossing  is  evident  (an  existing  bridge  and  a pontoon  bridge  in  construction). 
The  pilot  may  choose  to  designate  the  region  showing  several  strong  radar 
returns  and  thereby  slew  his  FLIR  to  that  direction.  One  of  the  simplest 
techniques  for  extracting  salient  FLIR  information  and  presenting  it  on  the 
radur  display  is  to  threshold  the  FLIR  output  and  use  the  symbol  generator 
to  place  a single  small  dot  at  the  centroid  of  each  hot  spot  detection.  Fig- 
ure 13b  shows  each  such  hot  spot  as  a red  dot  for  cuing  purposes.  On  a 
monochrome  display  without  multicolor  capability,  a bright  white  flashing 
dot  could  be  used  or  a shape-coded  symbol. 

Other  sensor  information  can  also  be  displayed  by  means  of  symbolog 
The  yellow  diamond  represents  an  emitter  whose  coordinates  have  been  data- 
linked  to  the  aircraft.  A green  triangle  could  be  used  to  represent  a radar 
ground  moving  target  indication. 
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Assuming  the  pilot  then  designates  the  target  of  interest  with  his 
movable  cursor,  the  coordinates  are  automatically  entered  into  the  mission 
computer,  and  he  descends  rapidly  for  low  altitude  ingress.  During  run-in, 
the  target  position  relative  to  the  aircraft  is  continually  updated  by  the  com- 
puter, aided  by  the  navigation  systems.  If  pop-up  occurs  at  less  than 
50,  000  feet  slant  range,  the  pilot  typically  chooses  to  go  directly  to  his 
imaging  sensor  (for  target  reacquisition)  which  is  usually  a FLIR.  (All 
examples  shown  represent  cases  in  which  the  pilot  must  use  his  multipurpose 
display  rather  than  direct  visual/HUD  due  to  visibility,  nighttime,  etc.  ) 

Since  detection  and  recognition  are  normally  accomplished  with  the  narrow 
field-of- view  mode  (approximately  1 degree  to  3 degrees),  it  is  critical  that 
the  FLIR  be  automatically  slewed  and  accurately  pointed  in  the  right  direction. 
Should  the  target  not  be  within  the  field  of  view,  the  pilot  must  slew  the  FLIR 
manually  or  return  to  the  radar  for  redesignation  of  the  target. 

The  display  shown  in  Figure  13c  is  one  technique  for  going  directly 
to  an  EO  display  at  pop-up  while  maintaining  a radar  border  for  context  in 
case  the  entire  radar  display  must  be  recalled  for  target  redesignation.  In 
this  case,  the  scale  factors  of  the  EO  image  and  the  radar  are  obviously  very 
different.  Only  one  point  is  correlated  between  the  EO  and  radar,  that  is, 
the  single  point  designated  by  the  pilot  before  descent.  The  error  in  this 
correlation  is  primarily  a function  of  the  errors  introduced  in  target  position 
update  during  run-in.  If  the  target  designated  were  toward  the  edge  of  the 
display,  the  EO  image  would  also  be  located  at  that  position.  If,  in  this 
example,  the  pilot  could  not  reacquire  and  was  required  to  return  to  the  radar 
image,  the  presence  of  the  river  within  both  sides  of  the  radar  border  might 
facilitate  the  necessary  visual  reorientation. 

This  insert  technique  for  maintaining  radar  context  can  also  be  used 
with  other  radar  formats.  Figure  14a  shows  a doppler  beam  sharpened  radar 
ground  map  in  a PPI  format.  Preliminary  verification  of  target  complexes 
could  be  achieved  by  briefly  overlaying  stored  cartography,  as  shown  in 
Figure  14b,  on  the  radar  ground  map.  After  designation  with  a movable 
cursor  of  the  region  showing  indications  of  storage  tanks,  the  FLIR  is  auto- 
matically slewed  to  that  direction.  The  FLIR  image  appears  on  the  radar 
display  centered  at  the  point  of  designation,  as  shown  in  Figure  14c.  This 
simulated  combination  represents  400  FLIR  lines  inserted  in  an  875-line 
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a PRIMARY  SENSOR  SAR  (10  FT  RESOLUTION! 


b PRIMARY  SENSOR  SAR 

SECONDARY  SENSOR  FUR  HOT  SPOTS 
SUPPLEMENTARY  DATA  EMITTER  LOCATION 


c PRIMARY  SENSOR  EO 


SECONDARY  SENSOR  SAR  ' BORDER"  TO  MAINTAIN 


CONTEXT  ANO  ORIENTATION  FOR  POSSIBLE  RETURN 


TO  RADAR  IMAGE 


FIGURE  13  SIMULATED  SIMULTANEOUS  PRESENTATIONS  USING  SAR  AND  EO  SENSOR  FOR  PRELIMINARY  TARGET 
ACQUISITION  AND  RE  ACQUISITION  AFTER  POP  UP 
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a.  DBS  radar  ground  map. 


b.  Stored  cartography  corresponding  to 
DBS  ground  map. 

Figure  14.  Target  acquisition  using  radar  and 
FI. lit  imagery.  Case  2.  (Sheet  1 of  2) 
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i.  FUR  insert  on  DBS  map,  first  target  complex. 


d.  F LIR  insert  on  DBS  map,  second  target  complex. 

Figure  14.  Target  acquisition  using  radar  and 
FLIR  imagery,  Case  2.  (Sheet  2 of  2) 
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radar  display.  Should  this  not  be  the  desired  target,  the  FUR  can  be  manually 
slewed  and  the  FUR  insert  moved  correspondingly,  or  the  pilot  may  return 

I to  radar-only  for  redesignation  of  a second  potential  target.  Assuming  he 

now  designates  the  storage  tanks  at  the  upper  left  of  the  radar  display  (which 
are  visible  in  the  radar  image  of  Figure  14c  even  with  the  FLIR  insert  present 
and  verified  with  Figure  14b),  the  resulting  display  would  appear  as  in  Fig- 
ure 1 4d.  Because  the  designated  point  is  near  the  display  edge,  fewer  FLIR 
lines  (about  300)  are  displayed.  However,  the  display  of  the  FLIR  FOV  could 
be  centered  on  the  radar  display  if  desired,  irrespective  of  the  position  of 
the  designated  point. 

With  the  insert  type  of  simultaneous  presentation,  flashing,  shape- 

( coded  or  color-coded  symbology  representing  threats,  GMTI,  etc.,  could  be 

overlaid  on  either  the  radar  or  the  FLIR  imagery.  However,  placing  them 
on  the  radar  imagery  would  provide  cuing  information  for  subsequent  cursor 
designation  and  pointing  of  the  FLIR  sensor. 

FLIR  and  Radar 

At  ranges  of  less  than  10  nmi,  an  electro-optical  imaging  sensor  such 
as  FLIR  or  TV  is  usually  the  primary  sensor.  Just  as  the  FLIR  may  be 
helpful  in  cuing  the  pilot  to  a point  on  the  radar  display  for  longer  range 
target  acquisition,  so  the  radar  could  possibly  provide  cuing  information 
on  the  FLIR  display  for  shorter  range  target  acquisition.  Using  the  imagery 
in  Figure  15a  to  simulate  a FLIR  or  TV  sensor  display,  we  see  a wide  and  a 
narrow  canal,  a wide  double  road  or  highway,  two  narrow  roads  and  a highway 
intersection.  Figure  1 5b  is  a low  resolution  (80  ft.  ) SAR  image  of  the  same 
scene.  By  photographic  processing  techniques,  the  radar  glints  were 
extracted  from  the  SAR  image  and  color-keyed  in  green,  resulting  in  the 
composite  image  in  Figure  15c.  Information  which  has  now  been  added  to  the 
FLIR  or  TV  image  includes  the  indication  of  an  irrigation  lock  at  the  junction 
■ f the  tw'o  canals  and  the  possible  presence  of  one  or  more  vehicles  at  the 
ghway  crossing.  Were  this  a true  FLIR  image,  the  presence  of  white  /hot 
I o k /hot)  objects  at  the  crossing  would  further  substantiate  the  presence 
■ - g vehic  les.  The  extended  radar  returns  at  the  left  are  reflections 

« r ' iad  bank . 

>’  ■ . rring  back  to  the  scene  insert  in  Figure  13c,  the  use  of  TV  as  the 
■ • - i tr  provide  visual  detail  of  vehicles  and  the  use  of  FLIR  as 
an  provide  thermal  indications  of  moving  or  still-warm 
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vehicles.  However,  neither  sensor  is  likely  to  detect  a cold  vehicle  in  foliage. 
The  use  of  amplitude-thresholded  radar  returns  or  glints  superimposed  on  the 
FLIR,  for  example,  could  cue  the  pilot  to  a cold  vehicle  or  man-made  target, 
or  verify  the  presence  of  a hot  or  moving  vehicle.  A radar  moving  target 
indication  displayed  as  a symbol  would  provide  further  verification  of  a mov- 

Iing  vehicle.  It  would  also  be  desirable  to  associate  particular  colors  with 

particular  sensors,  if  color  coding  is  utilized,  such  as  green  for  radar  glints 
and  red  for  FLIR  hot  spots. 

Symbols  can  be  overlaid  on  the  FLIR  or  TV  azimuth-elevation  dis- 
play for  anticipated  target  position,  laser  spot  tracker  coordinates,  laser 
designator /ranger  coordinates  (numerical  range  could  be  presented  adjacent 
to  the  symbol),  threat  location  from  warning  receivers,  a priori  or  data- 
linked  threat  locations,  and  moving  target  indication  or  fixed  target  track 
position. 

Case  3 

This  case  pertains  to  minimum  range  pop-up  after  run-in  to  target. 

By  maintaining  a low-altitude  penetration  profile  until  very  close  to  the 
target  (previously  cued  or  acquired),  the  aircraft  can  take  maximum  advan- 
tage of  terrain  masking  from  ground  defensive  systems,  thereby  reducing 
its  susceptibility  to  enemy  detection  and  attack.  However,  pop-up  at 
minimum  range  also  means  the  pilot  has  less  time  for  target  re- acquisition 
and  weapon  delivery,  with  timelines  possibly  as  short  as  10  seconds. 

It  is  assumed  that  the  primary  sensor  at  these  ranges  is  an  EO 
imaging  sensor,  such  as  FLIR  or  TV,  which  has  been  suitably  cued  to  allow 
the  use  of  its  narrow  field  of  view  mode.  This  attack  mode  often  depends 
on  storing  the  target  coordinates  in  the  inertial  navigation  system  prior  to 
the  penetration  portion  of  the  attack  profile.  The  EO  FOV  must  then  be 
consistent  with  the  resulting  navigation  and  pointing  errors.  If  both  a FLIR 
and  a TV  sensor  were  available,  the  possible  augmentation  of  one  sensor  by 
the  second  should  be  considered  to  aid  the  operator  in  the  target  acquisition 
problem.  Since  each  sensor  provides  different  information  about  a target 
. (the  FLIR  image  is  based  on  apparent  temperature  differences  within  a 

i 

scene,  while  the  TV  image  is  based  on  differences  in  light  reflectivity),  the 
combination  of  the  two  sensor  images  in  some  optimum  fashion  may  result 
in  enhanced  target  acquisition  performance, 
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An  example  of  combined  FLIR  and  TV  imagery  from  the  imagery  data 
base  acquired  during  th<:'  study  is  shown  in  Figure  lb.  Figure  16  corresponds 
to  a scene  as  sampled  and  displayed  by  a TV  system.  Figure  16  corresponds 
to  the  same  scene  as  sampled  and  displayed  by  a FLIR  system.  As  indicated 
in  Section  4.  2,  the  TV  and  FLIR  have  similar  magnifications  and  their 
boresight  can  be  manually  adjusted.  Both  sensors  are  operating  simultane- 
ously and  their  video  can  be  mixed  with  variable  gain  control  of  each.  At 
0.5  gain,  for  example,  hot-bright  targets  will  be  double  the  intensity  of 
cold-bright  or  hot-dark  areas. 

Figure  16  shows  the  two  videos  mixed  and  only  slightly  misregistered. 
Several  observations  can  be  made  by  comparing  the  mixed  video  to  the  TV 
or  FLIR  video  only.  Firstly,  there  is  more  detail  available  in  the  mixed 
video.  Note  that  the  ship  is  only  a silhouette  in  the  TV  mode  due  to  lighting 
geometry.  Note  also  that  the  FLIR  only  mode  presents  the  ship  as  saturated 
due  to  the  relatively  large  thermal  gradient  from  background  to  ship.  Mixing 
of  the  same  TV  and  FLIR  videos,  without  any  processing  or  enhancement, 
allows  more  features  of  the  ship  to  be  resolvable,  as  evidenced  by  the  figure. 

Secondly,  the  mixed  video  represents  a more  complete  and  accurate 
set  of  information.  As  an  example,  note  in  Figure  16b  that  the  boat's  mast 
is  shortened  because  of  a signal  processing  anomaly  in  the  FLIR.  The  TV 
of  Figure  16a  shows  the  mast  at  its  correct  height;  thus,  the  mixed  video 
preserves  this  information. 

Thirdly,  the  subsets  or  local  areas  of  the  TV  and  FLIR  images 
generally  have  different  video  amplitudes  and  modulation  characteristics. 
Thus,  in  some  cases,  the  mixed  video  may  result  in  an  effective  "filling  in" 
of  low  modulation  areas  of  one  sensor  with  image  detail  from  the  second 
sensor. 

Fourthly,  it  can  be  seen  that  the  effects  of  image  misregistration  do 
not  necessarily  result  in  degraded  image  quality.  In  Figure  16c,  the  TV  and 
FLIR  are  slightly  misregistered.  In  Figure  16d,  they  are  more  grossly  mis- 
registered, yet  the  image  is  still  highly  recognizable  and  informative.  Note 
also  in  Figure  16c  an  edge  enhanced  (or  outlined)  and  sculptured  (or 
3-dimensional)  effect  due  to  the  combined  effects  of  the  slight  misregistration 
and  the  TV/FLIR  video  amplitude  differences.  This  effect  is  much  more 
pronounced  when  viewing  the  dynamic  mixed  imagery  on  the  display,  as 
compared  to  the  static  photographs  presented  here. 
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c.  TV  plus  FLIR  slightly  d.  TV  plus  FLIR  grossly 

misregistered.  misr  egistered. 

Figure  16.  Summation  of  TV  and  FLIR  video 
showing  effect  of  misregistration 
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An  important  corollary  to  the  above  observation  is  that  pixel-by-pixel 
registration  may  not  be  required  or  even  desirable  in  simultaneous 
presentations.  Should  this  be  the  case,  the  complexity  of  any  required  hard- 
ware and  software  associated  with  matching  and  mixing  video  from  different 
sensors  would  be  considerably  reduced.  Because  of  the  impact  of  this  con- 
sideration on  realizing  a fieldable  simultaneous  sensor  system,  additional 
effort  in  the  form  of  human  factors  measurements  is  warranted  to  determine 
the  allowable  tolerances  in  mixing  TV/FLIR  as  well  as  F LIR /radar. 

The  mixed  video  of  Figure  16  involved  a simple  summation  of  the 
FL1R  and  TV  signals.  However,  a number  of  more  sophisticated  processing 
and  mixing  techniques  could  be  considered  for  future  evaluation  including: 

• Interlacing  of  the  two  video  fields. 

• Alternation  of  the  two  images  w'ith  variable  frequency. 

• Thresholding  of  one  video  signal  and  placement  of  only  the  high- 
lights on  the  second  video  image.  FLIR  hot  spots  could  be  placed 
on  the  TV  display  or  TV  bright  spots  placed  on  the  FLIR  display. 

• Edge- enhancement  of  one  video  image  and  placement  of  only  the 
outlines  on  the  second  video  image.  Temperature  differences 
from  the  FLIR  scene  could  appear  as  bright  outlines  on  the  TV 
display  or  brightness  differences  from  the  TV  scene  as  bright 
outlines  on  the  FLIR  display. 

• Insertion  of  TV  video  where  the  FLIR  saturates  to  provide  detail 
when  the  FLIR  dynamic  range  is  greater  than  the  display 
dynamic  range. 

• Insertion  of  FLIR  video  where  the  TV  is  dark,  again  to  com- 
pensate for  the  limited  dynamic  range  of  the  display  or  to  fill 
in  detail  where  there  is  none. 

• Sampling  of  the  modulation  in  the  FLIR  scene  and  insertion  of 
TV  video  wherever  there  is  no  detail  in  the  FLIR  image,  or 
vice  versa. 

• Color  coding  of  any  of  the  above  combinations,  preferably  using 
red  for  FLIR  information  and  green  or  blue-green  for  TV. 

Several  of  the  above  techniques  could  also  be  applied  to  the  FLIR 
display  only  (or  the  TV  only)  such  as  amplitude  thresholding  to  accentuate 
hot  (or  bright)  spots,  edge  enhancement  to  outline  areas  of  constant  tem- 
perature (or  brightness),  or  assigning  different  colors  to  different  level 
slices  such  that  the  dynamic  range  of  the  display  is  artificially  enhanced 
by  the  use  of  false  or  pseudocolor. 
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4.4  CONCLUSIONS 


To  determine  the  most  logical  simultaneous  sensor  display  candidates 
within  the  structure  of  the  interdiction  mission,  the  selection  criteria  used 
were:  1)  criticality  of  the  various  sensors  and  data  to  the  mission  and 

2)  utility  to  the  operator  of  combining  those  sensors  or  data  on  a single 
display.  Because  very  little  attention  has  been  previously  concentrated  in 
this  area  (as  demonstrated  by  the  literature  search  in  Appendix  A)  and  due 
to  the  exceedingly  limited  amount  of  existing  coincident  coverage  multisensor 
imagery,  the  values  assigned  to  criticality  and  utility  here  are  qualitative 
estimates.  Nonetheless,  definite  recommendations  emerge.  The  following 
simultaneous  sensor  presentations  appear  to  offer  significant  advantages 
for  increasing  the  probability  of  mission  success  and  for  decreasing  the 
operator  task  loading: 

• The  simultaneous  display  of  real-time  radar  ground  map  and 
stored  cartography  and  waypoint  data  to  assist  in  accurate 
navigation  to  the  prebriefed  target  area. 

• The  simultaneous  display  of  real-time  radar  ground  map  and 
stored  reconnaissance  imagery  of  prebriefed  targets  for  pre- 
liminary target  acquisition  at  long  range,  for  target  verification, 
and  for  offset  aimpoint  designation. 

• The  simultaneous  display  of  real-time  radar  ground  map  and 
real-time  FLIR  data  at  stand-off  ranges  to  provide  thermal 
gradient  indications  for  cuing  the  operator  to  possible  targets 
on  the  radar  display.  This  could  be  particularly  valuable  for 
the  acquisition  of  large  target  complexes  or  for  hot  spot  detec- 
tions corresponding  to  targets  of  opportunity  such  as  moving 
vehicle  s. 

• The  simultaneous  display  of  real-time  FLIR  imagery  and  radar 
glints  for  cuing  the  operator  to  man-made  objects  on  the  FLIR 
display  which  could  be  cold  or  hot. 

• The  simultaneous  display  of  real-time  FLIR  and  real-time  TV, 
processed  and  mixed  in  an  optimum  fashion  such  that  certain 
types  of  targets  may  be  acquired  in  shorter  time  or  at  longer 
range. 

Based  on  a qualitative  evaluation  of  the  simultaneous  presentations 
which  were  simulated  from  a very  limited  imagery  data  base,  the  following 
observations  concerning  the  display  of  these  data  can  be  made: 

• The  simple  summation  of  imagery  from  two  sensors  may  result 
in  lower  performance  than  could  be  achieved  with  one  sensor 
only.  A major  factor  in  achieving  a synergistic  combination 
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of  multisensor  data  is  the  selective  extraction  of  salient 
information  from  one  sensor  image  for  superposition  on  the 
second  sensor  image.  Thus,  the  key  to  effective  multisensor 
integration  and  display  is  image  processing. 

• Of  the  various  image  processing  operations  considered  in  this 

study,  two  techniques  show  particular  promise:  1)  thresholding 

of  one  sensor  image  and  subsequent  display  of  this  highlight  data 
on  a second  sensor  image  (for  example,  F LJR  hot  spots  in  red 
on  radar  ground  map  in  black/white  or  green,  FLIR  hot  spots  in 
red  on  TV  imagery  in  black/white  or  blue-green),  and  2)  edge 
enhancement  of  one  sensor  image  and  subsequent  display  of  these 
outlines  on  a second  sensor  image  (for  example,  FLIR  outlines 
of  roads  and  high  thermal  gradient  objects  displayed  on  radar, 

TV  outlines  of  high  contrast  objects  displayed  on  FLIR).  Special 
processing  operations  should  be  considered  to  ensure  that  essen- 
tial information  from  one  sensor  is  not  masked  or  replaced  by 
highlight  information  from  the  other  sensor. 

• As  more  sophisticated  image  processing  techniques  evolve  for 
feature  extraction  and  pattern  recognition  of  roads,  rivers,  tanks, 
trucks,  etc.  , from  typical  sensor  imagery,  provision  should  be 
made  to  display  the  results  of  these  automatic  feature  and  target 
cuing  devices  (for  example,  as  coded  symbols  or  graphics 
superimposed  on  FLIR  or  radar  imagery). 

• Exact  pixel  by  pixel  registration  of  two  superimposed  sensor 
images  may  not  be  required  or  even  desirable.  FLIR/TV  com- 
binations have  shown  that  slight  misregistration  can  result  in 
edge  enhancement  and  ’’sculpturing"  of  the  imaged  target.  By 
reducing  the  registration  requirements,  the  complexity  of  the 
har dware / software  required  to  register  and  mix  two  sensor  video 
could  also  be  substantially  reduced. 

• Although  all  of  the  simultaneous  presentations  could  be  presented 
on  a black/white  or  single  color  display,  color  coding  of  the 
different  sensor  inputs  appears  to  be  more  effective  in  that  it 
enables  the  observer  to  easily  differentiate  between  data  sources. 

• Associating  a specific  color  with  a specific  sensor  seems  highly 
desirable,  especially  for  highlight  data.  Examples  are  red  for 
FLIR  hot  spots,  green  for  radar  glints. 

• Red  does  not  appear  to  be  an  appropriate  color  for  background  or 
primary  sensor  imagery,  but  is  excellent  for  presenting  highlight 
data  from  another  sensor  or  for  overlaying  threat  symbology  on 
radar  or  FLIR  imagery. 

Future  efforts  should  include:  1)  the  ac  quisition  of  appropriate  simul- 

taneous, coincident  coverage  multisensor  imagery,  2)  the  spatial  warping  and 
registration  of  that  imagery,  3)  the  application  of  a variety  of  image  proces- 
sing operations,  and  4)  the  capability  to  display  the  registered,  processed 
imagery  in  a variety  of  color  combinations.  Thus,  quantitative  measurements 
of  operator  performance  could  be  made  as  a function  of  image  misregistration, 
image  processing  or  enhancement  technique,  and  false  color  selection. 
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5.  0 IMAGE  PROCESSING  CONSIDERATIONS  FOR 
SIMULTANEOUS  SENSOR  PRESENTATIONS 


A number  of  promising  simultaneous  presentations  were  identified 
in  Section  4.  0 for  use  in  an  interdiction  mission.  Implementation  of  these 
various  simultaneous  presentations  requires  that  the  two  (or  more)  types  of 
information  to  be  combined  first  be  made  compatible  (preprocessed)  for 
image  superposition,  then  optimally  processed  and  mixed  for  subsequent 
display  on  a single  monitor.  This  section  discusses  the  preprocessing  and 
processing  requirements  for  simultaneous  presentations,  such  as  refor- 
matting of  the  two  input  images,  correcting  for  image  misregistration, 
enhancing  or  selecting  salient  features  in  either  image,  and  displaying  the 
two  images  simultaneously.  Section  6.  0 will  then  describe  the  hardware 
required  to  implement  these  processing  operations. 

Three  types  of  simultaneous  presentations  have  been  recommended 
as  promising  candidates: 

1.  Real-time  sensor  plus  stored  cartography, 

2.  Real-time  sensor  plus  stored  reconnaissance  imagery, 

3.  Real-time  sensor  plus  real-time  sensor  such  as  radar  plus 
FLIR  (or  TV)  highlights,  FEIR  (or  TV)  plus  radar  highlights, 

FLIR  plus  TV,  etc. 

The  simplest  case  is  that  of  combining  primary  sensor  video  and 
cartography,  where  key  features  such  as  roads,  rivers,  coastlines,  etc.,  are 
available  for  matching  and  superposition. 

A more  difficult  undertaking  is  the  combination  of  real-time  sensor 
video  and  a priori  imagery  stored  in  the  aircraft  data  base  (or  imagery 
data-linked  in  flight),  requiring  the  correlation  of  two  images  which  may  have 
different  aspect  angles,  scale  factors,  and  scene  characteristics. 

The  most  complex  case  is  that  of  combining  the  video  from  two  real- 
time sensors.  In  general,  the  two  sensors  may  have  differences  in 

• Format  (example:  radar  is  azimuth  versus  range  while  FLIR 

is  azimuth  versus  elevation) 

• Line-scan  (example:  radar  may  have  875  TV  lines  and  FLIR  525 

or  vice  versa) 

• Field-of- view  (example:  radar  may  vary  in  azimuth  from  20  to 

120  degrees,  FLIR  FOV  from  1.  5 to  20  degrees,  and  TV  FOV 
from  0.5  to  40  degrees) 
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• Update  rate  (example:  radar  may  take  several  seconds  to  form 

one  frame  while  FLIR  is  updated  at  1/15  or  1/30  second  per 
frame,  and  TV  at  1/30  second  per  frame) 

• Orientation  (example:  two  images  may  have  errors  in  boresight 

and  rotation  (image  roll)). 

This  section  discusses  the  most  complex  case,  i.  e.  , the  processing 
requirements  for  combining  video  from  two  real-time  sensors.  The  same 
considerations  can  be  readily  applied  to  the  two  simpler  cases  involving  a 
real-time  sensor  and  stored  or  data-linked  imagery,  or  cartography. 

The  processing  operations  fall  into  four  areas,  as  shown  in  the  system 
block  diagram  of  Figure  17: 

1.  Preprocessing  and  coordinate  transformations  of  the  input  sensors 
such  as  radar,  FLIR,  or  TV  (as  well  as  data-linked  imagery  or 
stored  imagery  from  the  aircraft  data  base), 

Z.  Determining  and  correcting  misregistration  between  images.  (A 
registration  or  boresight  computer  determines  the  relative  shifts 
between  selected  sensor  imagery  and  provides  outputs  for  pre- 
detection or  po st- detection  correction  of  the  sensor  data.  ) 

3.  Applying  image  processing  and  enhancement  operations  to  either 
image , 

4.  Combining  the  processed  video  from  the  sensor  (or  stored/ linked 
imagery)  for  input  to  a display. 
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These  four  operations  are  described  below.  A number  of  possible  correction 
and  processing  operations  are  described  for  the  sake  of  completeness  and  to 
indicate  what  a highly  sophisticated  system  could  include  for  greatly  enhanced 
capability.  In  practice,  however,  high  performance  operational  systems  can 
be  implemented  utilizing  only  a selected  number  of  the  possible  functions 
identified  here  and  often  with  far  less  stringent  accuracies  than  these  functions 
are  capable  of. 

5.  1 PREPROCESSING 

The  signal  preprocessing  step  immediately  follows  the  signal  acquisi- 
tion from  the  sensors.  The  sensor  video  data  are  digitized  with  A/D  con- 
verters; thus,  two  A/D  converters  are  required,  as  a minimum.  The 
multiplexer  shown  in  Figure  17  controls  the  inputs  to  the  A/D  converter  and 
selects  from  various  inputs  to  be  displayed.  The  microprocessor-controlled 
scan  converter  is  an  important  element  in  reformatting  the  image  line  and 
scan  parameters  to  a common  standard.  Various  error  correction  inputs 
from  the  registration  computer  also  enter  here  for  horizontal  and  vertical 
scaling  and  shift.  All  the  correction  functions  are  closely  interrelated,  so 
the  preprocessing  steps  shown  in  the  initial  boxes  may  be  combined  in  an 
actual  system.  The  following  sections  consider  the  details  of  various  possible 
corrections  which  could  be  performed  if  required. 

5.1.1  Coordinate  Transformation 

A reformatting  of  one  sensor  input  may  be  required  to  achieve 
coordinate  compatibility  with  the  second  sensor  input.  For  example,  the 
display  of  FLIR  hot  spots  on  a radar  display  requires  the  transformation  of 
the  azimuth  elevation  coordinates  of  the  FLIR  data  to  the  azimuth- range 
coordinates  of  the  radar  image.  Thus,  a first  order  warp  could  be  used  in 
which  the  FLIR  image  plane  (perpendicular  to  the  sensor  line  of  sight)  is 
warped  into  the  radar  footprint,  based  on  own-aircraft  altitude  and  FLIR 
depression  angle.  This  coordinate  transformation  could  be  accomplished 
as  the  data  are  entered  into  the  scan  converter  memory.  The  simplest 
technique  would  be  to  command  the  symbol  generator  to  place  a dot  on  the 
radar  display  at  the  coordinate-transformed  location  of  each  thresholded  hot 
spot  detection. 
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The  placement  of  radar  glint  data  on  FLIR  imagery  would  require  a 
warping  of  the  radar  footprint  into  the  FLIR  imaging  plane.  This  computation 
is  the  inverse  of  that  recjuired  to  warp  the  FLIR  into  the  radar. 

5.1.2  Geometrical  Distortion 

In  the  most  general  form,  geometrical  distortion  can  be  visualized  by 
considering  a rubber  sheet  stretching  analogy.  If  a rectangular  grid  is 
applied  to  a flexible  sheet,  then  arbitrary  stretches  of  the  sheet  will  distort 
the  grid  to  a series  of  intersecting  curved  lines.  These  distortions 

can  be  caused  by  imperfections  in  the  imaging  system,  nonlinear  deflection 
in  the  image  sensor,  or  elevation  differences  in  the  observed  scene.  The 
fixed  sensor  distortions  (such  as  pincushion  or  barrel  distortion)  are  relatively 
small  and  vary  slowly  with  time.  They  can  be  removed  before  image  detec- 
tion by  compensation  of  the  deflection  which  is  generally  done  by  analog 
techniques  for  simplicity. 

In  the  case  of  elevation  or  terrain  height  differences,  certain  local 
dynamic  scene- dependent  distortions  may  occur  in  an  image  from  hills, 
mountains,  buildings,  etc.  , in  the  field-of- view.  Dynamic  correction  of 
these  distortions  is  very  difficult  because  the  scene  changes  rapidly  and  a 
fast,  sophisticated  processor  needs  to  continuously  correlate  frames  of 
information  and  provide  local  distortion  update  corrections.  It  is  not 
apparent  that  this  correction,  although  desirable,  is  necessary  for  this 
application.  Although  no  details  are  described  here,  this  type  of  correction 
is  feasible  and  extremely  useful  for  terrain  avoidance  close-range  systems. 

It  is  possible  that  the  use  of  an  area  correlation  technique  for  registering  two 
sensor  images  may  automatically  provide  a degree  of  correction  for  local 
dynamic  distortions,  thus  relieving  some  of  the  difficulties  and  complications 
of  this  case.  This  consideration  is  beyond  the  scope  of  this  contract,  and 
no  further  discussion  will  be  given  here. 


2A.  A.  Sawchuk,  "Correction  of  Imaging  System  Geometrical 
Nonlinearities  in  Real  Time,  " submitted  to  IFEE  Transactions  on 
Computers. 
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5.1.3  Intensity  Nonlinearity 

The  signal  output  of  an  image  sensor  may  not  be  a strictly  linear 
function  of  the  brightness  at  a point  in  the  scene. ^ In  addition,  there 
are  variable  position-dependent  shading  effects;  most  sensors  are  brightest 
in  the  center  and  drop  off  toward  the  edges.  Some  of  these  effects  can  be 
compensated  by  optical  design  and  by  analog  electronic  techniques.  Any 
residual  effects  could  be  compensated  digitally,  if  necessary,  by  the  use  of 
a fixed  or  variable  function  memory  or  programmable  read-only  system 

that  produces  a corrected  gray-level  value  from  inputs  (gamma 

, (Ref.  4)  T ...... 

correction).  In  many  cases,  linear  behavior  is  not  necessarily 

desired,  and  contrast  enhancement  may  be  included  at  later  stages  of 

processing.  As  a less  precise  but  simpler  and  more  rapid  technique, 

position-dependent  correction  can  be  applied  to  many  adjacent  pixels 

simultaneously.  This  avoids  excessive  hardware;  however,  care  must  be 

taken  to  avoid  blockiness.  A general  circular  symmetry  is  useful  for  this 

application  because  the  general  two-dimensional  x,  y variation  is  reduced 

to  a single  radial  variable. 

5.2  IMAGE  REGISTRATION 

After  the  geometrical  and  intensity  non-linearities  have  been 
sequentially  corrected  to  their  required  accuracies,  the  sensor  information 
goes  to  a registration  computer  which  determines  the  amount  of  misregis- 
tration and  supplies  signals  to  other  systems  for  correction.  Some  of  the 
possible  sources  of  mi s reg i stration  are  identified  here  and  possible  cor- 
rection techniques  are  described. 

5.2.1  Sources  of  Misregistration 

In  typical  systems,  several  types  of  misregistration  sources  are 
usually  present.  In  general,  it  is  possible  to  sequentially  correct  for  many 


A.  A.  Sawchuk,  "Real-Time  Correction  of  Intensity  Nonlinearities  in 
Imaging  Systems,"  IEEE  Transactions  on  Computers,  vol.  C-2b, 
January  1 977. 

4 

A.  R.  Colgan,  "Integrated  Cockpit  Display  System,  " Tech.  Report 
AEAL-TR-73-50,  Air  Force  Avionics  Laboratory,  (AFAL/AAA), 

W right- Patter  son  Air  Force  Base,  Ohio  45433,  January  1 973. 
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of  them  because  none  of  them  interact.  It  is  assumed  that  geometrical  and 
intensity  nonlinearities,  which  are  particularly  dependent  on  the  individual 
sensors  and  do  not  generally  interact,  have  already  been  corrected  to  a degree 
consistent  with  acceptable  tolerances  for  that  sensor  and  system  application. 

One  problem  in  image  registration  is  that  the  two  images  to  be  regis- 
tered are  similar,  but  not  identical,  representations  of  the  same  general 
scene.  Thus,  registration  is  possible  only  if  there  are  features  of  both 
images  that  are  sufficiently  identical  to  be  matched.  While  gray  scale  values 
in  LLLTV  and  FLIR  images  will  not  be  the  same,  certain  dominant  features 
such  as  roads,  rivers,  man-made  structures,  etc.  , should  be  apparent  from 
each  image.  With  this  correspondence,  it  is  possible  to  estimate  other 
residual  types  of  misregistration.  The  problem  becomes  more  difficult  when 
registering  radar  and  FLIR  or  radar  and  optical  images.  In  these  cases, 
few  features,  other  than  a river  or  a characteristic  pattern  such  as  an  air- 
strip, may  be  readily  discernible  in  each.  However,  the  fact  that  both 
sensors  are  imaging  the  same  scene,  within  the  limitations  of  pointing 
accuracy,  should  facilitate  the  task. 

Sensor  noise  or  processing  peculiarities  (such  as  translations  due  to 
processed  doppler)  may  prove  to  be  a limiting  factor  in  the  performance  of 
image  legistration  systems,  especially  at  very  low  signal  levels.  Other 
signal  processing  and  filtering  should  be  used  to  make  the  video  input  and 
geometric  and  brightness  corrections  prior  to  the  registration  computer  as 
noise-free  as  possible. 

If  there  are  magnification  differences  between  various  sensors,  an 
object  may  have  a different  size  within  each  scene.  Even  if  optical  magni- 
fication is  controlled,  there  may  be  electrical  magnification  differences 
between  scanners.  The  effect  could  also  be  dynamic  if  zoom  capabilities 
were  included  in  the  system. 

There  may  be  a horizontal  or  vertical  translation  difference  between 
objects  in  two  frames.  This  could  result  from  mechanical  misalignment  or 
electrical  deflection  drift. 

Images  may  also  have  a rotational  misalignment  due  to  mechanical 
errors,  different  degrees  or  orders  of  freedom  of  the  sensor  gimbals, 
motion  of  the  sensor  mounts,  or  electrical  problems. 

f 
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5.2.2  Pete r minat ion  of  Registration  Errors 


In  attempting  to  compile  the  total  misregistration  from  all  sources, 
it  is  important  to  separate  the  static  eftects.  which  may  be  compensated  in 
advance,  from  other  effects.  Gross  translational  and  rotational  differences 
may  be  removed  by  calibration,  while  magnil ic ation  differences  or  other 
small  errors  can  be  removed  electronically  in  real-time.  Many  ot  the 
registration  errors  will  be  relatively  small,  perhaps  on  the  order  of  a few 
picture  elements  (pixels)  at  most.  This  will  greatly  reduce  the  search  time 
and  computational  effort  required. 

There  are  several  techniques  available  for  determining  the  misregis- 
tration between  image  pairs.  lRot‘  Horizontal  or  vertical  translation 

differences  are  the  easiest  to  find,  because  they  can  be  determined  by 
shifting  techniques  on  the  rectangular  picture-  grid.  Two  major  methods  of 
comparison  are  by  statistical  correlation  and  sequential  detection.  The 
sequential  methods  are  much  faster  at  eliminating  mismatches  because  not 
every  image  point  need  be  tested.  Finding  magnification  and  rotational 
differences  is  much  more  difficult,  because  of  the  rectangular  sampling  grid 
and  the  necessity  of  performing  these  distortions  on  the  images  to  be  regis- 
tered when  searching  for  a match.  Some  techniques  involving  transtorms 

, , (Ref.  9,  10) 

have  been  developed  which  tend  to  simplify  this  problem. 


"\j.  G.  Kavvamina.  ' Automatic  Recognition  of  Changes  in  Urban 
Development  from  Aerial  Photography."  IKKF.  Trans.  Systems,  Man 
and  Cybernetics,  vol.  SMC  - i,  1071,  p.  230. 

^R.  L.  Lillestrand.  "Technique s for  Change  Detection.  " IEEE  Trans- 
actions on  Computers,  vol.  C-21,  1972.  p.  654. 

'D.  I.  Bamea  and  H.  F.  Siberman,  "A  Class  of  Algorithms  for  Fast 
Digital  Image  Registration,  " IEEF.  Transactions  on  Computers, 
vol.  C-21.  Feb.  1972,  pp.  179-186. 

8VV.  K.  Pratt.  "Correlation  Techniques  of  Image  Registration.  " IEEE 
Transactions  Aerospace  and  Electronic  Systems. 

^H.  K.  Ramapriyan,  "A  Multi-level  Approach  to  Sequential  Detection 
of  Pictorial  Features.  " IFF.F  Transai  tions  on  Computers,  vol.  C-25, 
Jan.  1976.  pp.  66-78. 

1(^E.  L.  Hall,  "Almost  Uniform  Distributions  for  Computer  Image 
Enhancement.  " IEEE  Transactions  on  Computers,  vol.  C-23, 

Feb.  1974,  pp.  207-208. 


There  are  several  correlation  systems  in  development  at  Hughes 
Aircraft  Company  which  can  be  used  to  perform  the  registration  error 
computation.  An  area  video  correlator  has  been  developed  for  increasing 
the  accuracy  of  target  handoff  from  one  onboard  sensor  to  another,  or  to  a 
missile  sensor.  This  system  can  be  used  to  correlate  several  types  of 
sensor  video:  FL1R  with  FI.IR.  TV  with  TV,  and  FLIR  with  TV.  In  addi- 
tion, samples  of  radar  and  FLIR  imagery  have  been  successfully  correlated, 
indicating  that  the  technique  can  be  extended  to  be  applicable  to  general  FLIR 
and  high  resolution  radar  video. 

Although  the  area  correlator  is  a unique  and  straightforward  design, 
it  may  be  possible  to  utilize  an  even  simpler  device  such  as  an  n-line  corre- 
lator to  determine  the  image  registration  errors.  This  kind  of  correlator 
determines  translational  errors  between  two  sensor  images  with  respect  to 
a single  common  image  reference  point.  It  is  known  that  the  n-line  corre- 
lator is  somewhat  insensitive  to  rotational  errors.  However,  by  selecting 
two  reference  image  points  and  performing  an  n-line  correlation  at  each, 
common  vectors  in  the  two  image  fields  can  be  identified  and  the  angle 
between  them  measured.  This  angle  is  the  rotational  difference,  which  can 
then  be  applied  to  correct  the  error.  Similarly,  comparison  of  the  vector 
lengths  gives  the  magnification  difference.  Thus,  by  using  a relatively 
simple  boresight  correlator  and  making  measurements  on  several  image 
points,  the  four  remaining  major  registration  errors,  X and  Y translation, 
rotation,  and  magnification,  may  be  determined. 

5.2.3  Correction  for  Misregistration 

Once  the  misregistration  parameters  are  identified,  they  are  fed 
back  to  the  preprocessing  system.  A microprocessor-controlled  modular 
digital  scan  converter  (MDSC)  system  can  easily  perform  translational 
corrections  and  magnification  corrections  based  on  the  inputs  from  the 
registration  computer.  The  horizontal/vertical  shifts  are  used  to  offset  the 
readout,  and  the  magnification  correction  is  used  as  input  to  the  scan  line 
matching  system.  The  difficult  operation  is  rotational  correction,  which 
would  be  very  complex  if  performed  in  the  MDSC  due  to  the  non-rectangular 
readout  which  would  be  required.  The  rotational  correction  could  be  applied 
directly  to  the  sensor  electrically  or  mechanically.  The  same  technique 
could  be  used  with  a zoom  control  for  feedback  magnification  correction. 
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For  these  two  types  of  errors,  correction  by  sensor  realignment  seems  to  be 
the  simplest  and  most  cost-effective.  However,  for  low  update  rate  sensors 
or  data,  the  MDSC  can  accomplish  these  corrections  adequately. 

5.3  IMAGE  PROCF.SSING 

After  format  conversion  and  registration  are  accomplished,  there  are 
a number  of  ditferent  image  processing  techniques  that  can  be  applied  to  the 
images  before  they  are  combined  for  presentation  to  the  observer.  A limited 
number  of  these  possible  techniques  are  considered  here,  including: 

• Gain  control 

• Thresholding  of  one  v ideo  signal  and  placement  of  only  the  high- 
lights on  the  second  video  image:. 

• Thresholding  of  one  video  signal  at  either  saturation  level  or 
minimum  intensity  level  such  that  the  second  sensor  video  can 
be  inserted  in  these  regions. 

• Sampling  of  the  scene  modulation  in  one  sensor  image  and  inser- 
tion of  second  sensor  video  in  areas  of  low  detail. 

• Edge  enhancement  of  one  video  image  and  placement  of  only  the 
outlines  on  the  second  video  image. 

• Level  slicing  for  outlining  regions  with  a common  brightness  or 
color  coding  regions  with  a common  brightness. 

• Advanced  image  analysis  techniques  such  as  change  detection  to 
detect  moving  objects  and  pattern  recognition  to  recognize  types 
of  targets. 

The  individual  and  relative  values  of  these  processing  techniques  for 
target  acquisition  are  yet  to  be  determined.  Human  factors  measurements 
are  needed  to  quantify  the  amount  of  performance  improvement,  if  any,  that 
could  be  achieved  with  the  use  of  these  operations.  While  all  these  opera- 
tions are  considered  in  the  general  system  for  the  sake  of  completeness, 
in  practice  only  one  or  two  may  be  selected  for  incorporation  in  an  opera- 
tional system.  Note,  however,  that  Section  6.0  will  show  that  most  of  the 
operations,  other  than  change  detection  and  pattern  recognition,  are  suf- 
ficiently simple  to  implement  such  that  a highly  flexible  general  system  is 
quite  feasible. 

Many  of  the  above  image  processing  functions  are  most  applicable 
to  TV,  FLIR,  or  other  images  with  large  gray  level  content.  Several  of 
these  techniques  are  not  applicable  to  radar,  such  as  modulation  sampling 
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or  thresholding  for  fill-in  with  video  from  another  sensor.  Selection  of  the 
functions  can  be  manual  or  programmed  to  be  semiautomatic  or  automatic 
with  pilot  override.  Under  severe  task  loading  conditions,  little  or  no  opera- 
ator  involvement  should  be  required,  and  the  selection  of  processing  and 
display  techniques  should  be  as  automated  as  possible. 

Some  of  the  processing  techniques  ideally  assume  the  availability  of 
a high  resolution  color  display  in  the  ultimate  system,  although  it  is  realized 
that  such  displays  are  in  the  development  stage  and  not  yet  standard  aircraft 
equipment.  It  is  anticipated  that  the  use  of  false  color  to  distinguish  between 
multi  sensor  information  combined  on  a single  display  may  facilitate  the  per- 
ception and  utilization  of  that  information.  However,  most  of  the  enhancement 
techniques  can  also  be  effected  on  a monochrome  display.  The  improvement 
in  performance  associated  with  using  a multicolor  display,  compared  to  a 
monochrome  display,  should  be  determined  by  means  of  a human  factors 
laboratory  evaluation. 

The  various  processing  features  which  were  considered  and  their 
possible  enhancement  of  the  visual  display  are  described  as  follows. 

5.3.1  Gain  and  Function  Memory 

A gain  control  or  contrast  enhancement  can  be  automated  to  provide 
a maximum  dynamic  range  without  clipping.  Implementation  is  by  digital 
table  lookup.  The  gain  control  can  be  expanded  to  include  a look-up  table 
mapping  function  to  correct  for  nonlinear  intensity  distortions  described  in 
Section  5.  1.  If  a broad  range  of  gray  values  is  desired,  a real-time  his- 
togram measurement  system  could  be  incorporated  to  keep  track  of  the  gray 
level  distribution.  Then,  a nonlinear  mapping  could  be  applied  to  pixels  to 
uniformly  distribute  the  values  over  the  display  dynamic  range  (histogram 
equalization). 

5.3.2  Thre  sholding 

This  nonlinear  function  determines  the  level  at  which  signals  from 
one  sensor  will  be  clipped  and  overlaid  as  highlight  data  on  another  sensor 
image.  For  example,  when  radar  is  the  primary  sensor  display,  the  FLIR 
video  could  be  thresholded  such  that  only  the  hot  spots  are  superimposed 
on  the  radar  image  as  a target  cuing  or  verification  mechanism. 
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Thresholding  is  also  useful  for  radar  glint  information.  These  radar  glints 
could  be  placed  on  a FLIR  display  to  cue  the  pilot  to  cold  vehicles,  for 
example,  which  have  no  strong  thermal  signature  or  as  verification  of  hot 
metal  objects.  When  superimposing  FI.IR  and  TV,  the  thresholded  FLIR 
hot  spots  could  be  placed  on  the  TV  display. 

Thresholding  for  ''fill-in"  can  be  used  with  FLIR  and  TV  as  a means 
of  "extending"  the  display  dynamic  range  in  the  following  manner.  The  FLIR 
video  can  be  thresholded  such  that  all  FLIR  information  above  a certain  level 
(within  a saturated  area)  is  replaced  by  TV  information.  An  alternative  is 
to  threshold  the  TV  video  such  that  all  TV  information  below  a certain  level 
(dark  area)  is  replaced  by  FLIR  information. 

5.3.3  Scene  Modulation  Sampling 

Thresholding  can  be  applied  to  image  features  otner  than  luminance. 
An  "activity"  or  modulation  measure  can  be  computed  in  a small  moving 
window  of  pixels  (in  both  sensor  scenes)  by  measuring  signal  variance  about 
the  mean.  If  this  variance  exceeds  a threshold,  signifying  a lack  of  detail 
in  that  area  of  an  image,  data  from  another  sensor  can  be  inserted.  Such 
a system  can  thus  provide  the  maximum  detail  from  several  operational 
sensors  and  may  be  particularly  suitable  for  synchronized  FLIR  and  TV 
combination  s. 

5.3.4  Edge  Enhancement 

Linear  high-frequency  emphasis  or  differentiation  operators  are 
useful  for  enhancing  edges  within  an  image  and  thus  assisting  in  shape  recog- 
nition. Edges  are  sometimes  displayed  as  binary  pictures  in  place  of  or 
together  with  the  continuous  tone  scene.  Because  they  are  linear,  however, 
they  are  somewhat  error-prone  and  noisy.  Recently,  several  higher  per- 
formance edge  operators  have  been  described  which  are  highly  nonlinear. 

One  of  the  simplest,  yet  effective,  is  called  the  Roberts  operator.  The 
Roberts  edge  detector  works  on  a 2 x 2 array  of  pixels.  Denoting  the  bright- 
ness of  two  adjacent  pixels  in  the  top  image  line  by  A and  B and  two  pixels 
just  below  in  the  next  line  by  C and  D,  the  Roberts  operator  output  is 

E = th  ||  A - D | + |B  - C | j 
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where  "th|-|  ''  denotes  a threshold.  The  absolute  values  are  easy  to  compute, 

and  the  computation  takes  only  two  lines  of  image  storage.  Other  versions 
of  edge  operations  are  available.  Some  operate  in  a 3 x 3 array,  and  others 
involve  complicated  edge  fitting  in  the  Fourier  domain. 

Edges  could  be  displayed  in  another  color  on  the  same  image  or  on  a 
second  sensor  image,  or  could  be  used  to  sharpen  the  edges  in  a continuous 
tone  picture.  Even  the  best  edge  detectors  suffer  from  noise,  which  leads 
to  fake  edge  points  and  gaps  in  object  outlines.  More  sophisticated  (and 
computationally  difficult)  techniques  involve  boundary  tracing,  or  connections 
between  edges  to  better  outline  objects  and  areas  of  interest.  More  study 
is  necessary  to  determine  the  relative  advantages  (if  any)  of  these  highly 
sophisticated  techniques  for  shape  recognition. 

5.3.5  Level  Slicing 

This  technique  can  be  used  to  outline  objects  with  a common  bright- 
ness. Often  this  display  technique  is  made  interactive,  and  the  "sliced" 
regions  are  displayed  at  high  luminance  or  in  a different  color.  By  providing 
multiple  thresholds,  a range  of  gray  level  values  can  be  color-coded.  For 
example,  the  sensor  dynamic  range  could  be  divided  into  three  regions  and 
a different  color  assigned  to  each.  In  the  extreme,  every  level  of  gray  can 
be  made  a different  color.  This  false  color  technique  makes  use  of  the  fact 
that  color  changes  can  be  more  easily  discriminated  than  small  gray  level 
changes  and  thus  causes  subtle  gray  level  changes  to  stand  out. 

5.3.6  Change  Detection 

This  feature  would  use  frame-to-frame  differences  for  the  detection 
of  moving  objects.  By  detecting  image  differences  or  using  more  sophisti- 
cated correlation  techniques,  movement  in  a fixed  time  period  could  be 
identified.  Since  most  moving  objects  in  a scene  are  man-made,  this  cuing 
information  could  be  extremely  useful  to  the  observer  for  target  acquisition. 

5.  3.  7 Automatic  Feature  Extraction  and  Target  Cuing 

High-level  image  processing  algorithms  can  be  used  to  analyze  com- 
ponent parts  of  the  scene  for  feature  extraction  and  pattern  recognition  of 
roads,  waterways,  bridges,  structures,  vehicles,  etc.  The  location  of 
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these  features  or  objects  might  be  indicated  by  a complex  graphics  or  symbol 
overlay  on  the  display  as  a target  cuing  mechanism.  Many  of  these  tech- 
niques for  object  pattern  recognition  and  target  cuing  are  presently  in  the 
research  stage.  The  recognition  and  extraction  of  complex  features  is  a 
difficult  task  requiring  a great  deal  of  computation.  The  decision  process 
consists  of  several  levels:  1)  feature  extraction  (such  as  edges,  texture, 

etc.),  2)  combination  of  these  features  into  higher  level  primitives  (shape, 
outlines,  segments  of  objects),  and  3)  interpretation  of  the  primitives  in 
terms  of  pattern  recognition.  Such  a system  would  be  valuable  for  reducing 
the  observer's  workload  directing  him  to  critical  areas  on  the  display.  It  is 
expected  that  substantial  progress  in  image  analysis  and  understanding  will 
be  made  in  the  future  and  that  the  tecnniques  developed  will  be  applied  to 
display  systems.  Thus,  prov  isions  for  display  of  the  extracted  data  should 
be  made  in  the  multi- sensor  display  system. 

5.4  COMBINING  IMAGES  FOR  DISPLAY 

This  section  describes  possible  techniques  for  combining  images 
from  two  or  more  sensors  on  a single  display.  The  selection  of  an  optimum 
combining  technique  will  require  a human  factors  experimental  evaluation 
of  the  candidates  listed  and  will  undoubtedly  depend  on  the  types  of  imagery 
being  mixed. 

5.4.1  Ar  ithmet  ic 

A simple  method  of  combining  multiple  sensors  is  to  use  a high  speed 
arithmetic  unit  which  produces  a single  output  at  each  pixel  from  multiple 
inputs.  A fixed  function  could  be  a weighted  arithmetic  or  geometrical  mean, 
or  a nonlinear  (logarithmic)  function  which  would  attempt  to  present  a maxi- 
mum useful  dynamic  range  to  the  observer.  Weights  on  the  means  could  be 
fixed  or  operator- variable,  or  could  be  automatically  varied  by  an  activity 
threshold  circuit. 

5.4.2  Interlac  e 

A more  direct  method  avoids  the  arithmetic  combination  and  inter- 
laces two  frames.  This  technique  would  require  additional  display  resolu- 
tion and  would  provide  the  same  visual  effect  as  averaging. 
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5.4.3  Inter  frame 


A flicker  mode  (alternate  frame  function)  could  be  used  to  switch 
from  one  sensor  to  another  at  a predetermined  rate  or  at  a threshold  con- 
trollable level.  Although  this  type  of  switching  is  straightforward,  the 
flicker  may  be  distracting  at  very  low  rates;  at  high  rates  ( >2 0 frames/ 
second),  the  human  visual  system  would  integrate  the  frames. 

5.4.4  Local  Area  Insertion 

The  insertion  of  one  sensor  image  (typically  narrow  field  of  view) 
within  a second  sensor  image  (wide  field  of  view)  represents  a possible 
technique  for  maintaining  WFOV  context  while  attempting  to  acquire  a target 
with  a NFOV  sensor.  An  example  is  the  designation  of  a possible  target  on 
the  radar  display  and  the  subsequent  display  of  a NFOV  F LIR  image  centered 
at  the  designated  point  while  a radar  image  "picture  frame"  border  is  main- 
tained for  context.  This  facilitates  the  return  to  the  radar  display  for 
redesignation,  if  the  NFOV  FLIR  scene  does  not  contain  the  desired  target. 
The  insertion  boundaries  may  be  manually  selected  or  automatically  selected 
by  an  image  cuing  system. 

5.4.5  Stereo 

Another  possible  alternative  would  make  use  of  the  height  discrimina- 
tion ability  of  stereo  vision.  Stereo  can  be  achieved  by  two  separate  dis- 
plays or  split  screen  on  a single  display  with  appropriate  optics  (goggles  or 
filters)  to  provide  separate  images  for  each  eye.  With  only  a monochrome 
system,  the  height  variable  could  provide  graphics,  or  highlights  or  glints 
from  one  sensor  could  be  displayed  as  a height  variable  on  top  of  a continuous 
tone  background.  The  stereo  possibility  has  received  little  attention  in  the 
past,  primarily  due  to  the  complexity,  but  it  may  be  advantageous  to  reevalu- 
ate some  of  the  methods  in  the  light  of  present-day  technology. 

5.4.6  Multicolor 

Although  simultaneous  presentations  could  be  combined  on  a black/ 
white  or  single  color  display,  the  use  of  false  color  coding  may  assist  in 
differentiating  information  or  imagery  from  several  sources.  It  would  thus 
be  desirable  to  associate  a particular  color  with  highlight  data  from  a 
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particular  sensor,  such  as  green  with  radar  glints,  red  with  F LIR  hot  spots 
and  blue  with  TV  bright  spots. 

It  is  anticipated  that  primary  sensor  or  background  imagery  would 
probably  be  displayed  in  a comfortable  broadband  white  or  yellow-green. 

For  example,  radar  imagery  could  be  displayed  in  green  while  FLIR  hot 
spots  or  edge-enhanced  features  could  be  overlaid  in  red.  FLIR  imagery 
could  be  displayed  in  broadband  white  while  radar  glints  are  overlaid  in 
green  or  TV  data  (object  outlines,  thresholded  bright  areas,  high-detail 
areas  for  FLIR  fill-in)  are  displayed  in  blue.  Color  can  also  be  useful  when 
overlaying  graphics  and  symbology  on  continuous  gray  level  imagery. 

As  described  previously,  false  color  coding  of  gray  level  "slices'' 
can  assist  in  enhancing  small  gray  level  changes.  Another  possibility  which 
may  merit  future  consideration  is  the  false  coloring  of  the  output  from  high 
level  feature  detectors  (based  on  texture,  gray  shade,  edges,  etc.  ).  Thus, 
for  example,  broad  regions  of  water  (identifiable  through  gray  level,  shape 
and  texture  characteristics)  might  be  colored  blue,  so  that  some  of  the  inter- 
pretive processing  typically  done  by  the  human  visual  system  is  now  per- 
formed by  the  proc  e s sor  / d i splay.  In  the  multi  sensor  application,  false  color 
could  be  used  to  color  code  regions  identified  from  several  sensors  by  a 
pattern  classifier.  In  this  way,  unique  spectral  signatures  acquired  from 
several  sensor  bands  could  be  used  to  present  a unique  color  to  the  observer. 
With  training,  an  observer  might  be  able  to  discriminate  among  very  subtle 
object  differences  based  on  all  the  sensor  inputs. 

5,5  CONCLUSIONS 

The  various  processing  operations  associated  with  accepting  two  (or 
more)  disparate  sensor  video  and  combining  them  on  a single  display  have 
been  discussed  in  this  section.  Four  general  functions  have  been  considered: 

1.  Preprocessing  operations  involving  coordinate  transformation 
and  line  scan  matching  to  make  the  two  video  format-compatible, 

2.  Determination  of  the  misregistration  between  the  two  images 
and  possible  correction  methods, 

3.  A shopping  list  of  image  processing  or  enhancing  techniques  for 
extracting  highlight  information  from  one  sensor  video  and  dis- 
playing it  on  a second  sensor  image. 

4.  A shopping  list  of  mixing  techniques,  including  the  use  of  false 
color  coding,  for  effectively  combining  information  from  two 
sensors  on  one  display  surface. 
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A sizable  number  of  possible  processing  operations  have  been 
included  here  for  completeness  and  for  indication  of  what  could  be  accom- 
plished if  desired.  In  practice,  however,  an  operational  system  may  require 
a very  limited  number  of  these  operations  to  achieve  significant  performance 
improvement.  Fine  corrections  may  not  be  required  for  local  geometric- 
distortions  or  brightness  nonlinearities.  Strict  tolerances  may  not  be 
required  for  image  registration;  in  fact,  slight  misregistration  of  images 
may  actually  enhance  target  acquisition.  As  few  as  one  or  two  simple  image 
enhancement  techniques  for  extracting  salient  information  from  one  sensor 
for  display  on  a second  sensor  image  may  enable  targets  to  be  acquired  at 
longer  range  or  in  shorter  time.  False  color  coding  of  highlight  information 
may  be  an  effective  means  of  assisting  the  operator  to  discriminate  between 
sensor  inputs. 

It  is  evident  that  an  experimental  evaluation  of  these  parameters  is 
needed  to  establish  minimum  system  requirements  and  to  quantify  changes 
in  operator  performance  as  a function  of  correction  tolerances,  enhancing 
operations,  and  color  combinations. 

It  is  also  evident  that  all  of  the  processing  operations  selected  for 
incorporation  in  an  operational  system  must  be  as  automated  as  possible.  On 
a prebriefed  mission,  the  sequencing  of  simultaneous  presentations  can  be 
preselected,  typically  as  a function  of  mission  phase.  Also,  the  processing 
operations  associated  with  each  sequential  simultaneous  presentation  can  be 
preprogrammed  or  made  highly  adaptive  and  automatic.  Thus,  operator 
interaction  and  task  load  can  be  minimized,  while  allowing  for  operator  over- 
ride in  all  cases. 
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6.  0 IMPLEMENTATION  CONSIDERATIONS  AND  TYPICAL 
SYSTEM  OPERATION 


This  section  discusses  the  conceptual  implementation  of  a general 
simultaneous  presentation  system  in  firms  of  hardware  and  processing 
requirements,  and  describes  typical  system  operation  for  several  sensor 
combinations. 

The  most  promising  simultaneous  presentation  candidates  which 
emerged  from  Section  4.  0 can  be  summarized  as: 

1.  Real-time  sensor  image  (radar)  and  stored  cartography, 

Z.  Real-time  sensor  image  (radar)  and  stored  or  data-linked  target 
imagery,  and 

3.  Two  real-time  sensor  images  (radar  and  FLIR,  FLIR  and  TV, 
etc.  ). 

The  most  complex  case,  i.  e.  , case  3,  was  addressed  in  Section  5.  0, 
where  a multiplicity  of  processing  operations,  both  required  and  optional, 
were  considered  for  the  simultaneous  display  of  two  real-time  sensors. 

These  same  considerations  could  be  readily  applied  to  the  two  simpler 
cases,  1 and  Z,  involving  a real-time  sensor  and  stored/data- linked  imagery 
or  cartography. 

As  the  next  logical  step,  the  complexity  of  implementing  the  most 
difficult  case,  the  combination  of  two  real-time  sensors,  will  be  considered. 
This  section  does  the  following: 

1.  Examines  the  pertinent  avionics  equipment  which  may  already 
be  onboard  the  aircraft, 

Z.  Describes  possible  implementation  techniques  for  the  various 
processing  operations  identified  in  Section  5.  0,  both  necessary 
and  optional,  thus  resulting  in  a general  system  implementation, 

3.  Describes  a typical  system  operation  for  two  complex  examples, 
i.  e,  , radar  plus  FLIR  and  FLIR  plus  TV,  and 

4.  Examines  the  key  elements  from  the  general  system  implementa- 
tion which  would  be  required  to  implement  each  of  the  promising 
simultaneous  presentations  identified  in  Section  4.  0. 

A concentrated  effort  has  been  made  in  the  implementation  consider- 
ations to  make  use  of  hardware  which  is  presently  available  or  in  develop- 
ment at  Hughes  Aircraft  Company.  This  use  of  state-of-the-art  equipment 
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whose  capabilities  and  interfaces  art'  fully  understood  was  considered 
preferable  for  sizing  the  mechanization  complexity,  as  compared  to  a more 
general  analysis  of  possible  alternate  implementation.  Thus,  the  following 
sequential  description  of  the  implementation  of  each  function  in  the  general 
system  is  based  on  Hughes'  related  engineering  experience  with  well- 
understood  technology. 

6.  1 ONBOARD  SENSORS/ PROCESSORS 

Several  promising  combinations  of  primary  sensors  and  supplementary 
sensors  or  stored  data  have  been  identified  for  simultaneous  presentation 
on  a multipurpose  display  as  a function  of  mission  phase.  Before  estimating 
the  complexity  of  implementing  these  combinations,  let  us  reexamine  the 
sensors  and  processing  equipment  which  are  assumed  to  be  onboard  the 
aircraft  and  the  manner  in  which  they  interface  for  each  of  the  selected 
combinations  or  mission  phases. 

The  baseline  sensors,  data  sources  and  associated  processors  con- 
sidered onboard  correspond  to  the  various  subsystems  which  comprise  the 
F-18  avionics  suite.  (This  suite  was  selected  as  representative  of  an 
advanced,  high  performance,  single- seat  attack  aircraft.  ) Figure  18  shows 
the  functional  interfacing  of  these  subsystems  for  key  mission  phases.  The 
corresponding  primary  and  secondary  sensors  or  data  sources  identified 
for  these  phases  are  as  follows. 

• During  cruise-out,  as  shown  in  Figure  18a,  the  primary  sensor 
is  the  radar  for  navigation  purposes,  supplemented  by  a priori 
data  such  as  stored  cartography,  stored  target  and  threat  data, 
and  threats  detected  by  onboard  warning  receivers. 

• Preliminary  target  acquisition  may  be  achieved  during  cruise 
and  before  descent  for  run-in.  As  represented  in  Figure  18b, 
the  radar  again  is  the  primary  sensor  and  can  be  supplemented 
by  a priori  data  such  as  stored  target  imagery,  by  stored  and 
real-time  warning  information,  and  possibly  by  hot  spot  cuing 
indications  from  a high  performance  FLIR. 

• Stand-off  range  pop-up  can  also  be  represented  by  Figure  18b. 

The  primary  sensor  in  this  case  depends  on  the  range.  For 
slant  ranges  greater  than  50,  000  feet,  radar  is  typically  the 
primary  sensor,  possibly  supplemented  by  FUR  hot  spots. 

At  ranges  less  than  50,000  feet,  FLIR  is  the  primary  sensor, 
possibly  supplemented  by  radar  glints. 

• At  minimum  range  pop-up,  as  shown  in  Figure  18c,  the  FLIR  is 
typically  the  primary  source  for  target  reacquisition  and  weapon 
delivery,  unless  severe  weather  conditions  prevail.  The  possible 
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Figure  18.  Interfacing  of  F-18  avionics  subsystems  as  a 
function  of  mission  phase. 
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correlation  of  missile  sensor  imagery  with  onboard  FLIR  imagery 
could  greatly  improve  the  accuracy  of  boresight  and  target 
hand-  of  f. 

The  following  subsection  describes  a general  implementation  which 
will  accept  inputs  from  any  of  the  above  sensors  and  data  sources,  will 
spatially  warp  and  register  any  two  desired  images,  will  perform  various 
selectable  enhancing  operations  on  one  or  both  images,  and  will  subsequently 
present  the  combined  imagery  on  a single  display.  It  is  anticipated  that 
portions  of  the  onboard  proc  essing  equipment  such  as  the  CPU  and  the 
optional  boresight  c orrelator  can  be  time-shared  for  use'  in  the  multisensor 
display  implementation. 

6.2  IMPLEMENTATION  CONSIDERATIONS 

This  section  describes  possible  methods  of  implementation  which 
includes  all  of  the  functions  shown  in  the  system  block  diagram  of  Figure  19. 
The  system  can  ac  cept  inputs  such  as: 

1.  real-time  radar,  F FIR  and  TV  video, 

2.  stored  or  data-linked  target  imagery  previously  acquired  from 
reconnaissance  cameras,  infrared  sensors,  radars,  etc., 

3.  stored  cartography  (to  be  displayed  as  graphics  and  symbology), 

4.  real-time  warning  receivers,  moving  target  indicators,  laser  spot 
trackers  and  rangers,  etc.  (to  be  displayed  as  symbology),  and 

5.  stored  or  data-linked  threat  data  (to  be  displayed  as  symbology). 

This  section  concentrates  on  the  implementation  of  the  most  complex 
combination,  i.  e.  , two  real-time  sensors  such  as  radar  and  FLIR  or  FLIR  and 
TV.  It  is  assumed  that  the  same  implementation  which  satisfies  this  case 
is  directly  applicable  to  the  simpler  cases  of  combining  a real-time  sensor 
such  as  radar  (Input  1 ) with  stored  imagery  (Input  2)  or  stored  cartography 
(Input  3).  Information  corresponding  to  Inputs  4 and  5 bypass  the  imagery 
processing  functions  and  go  directly  to  the  microprocessor  which  commands 
appropriate  symbology  to  be  generated  and  placed  on  the  display. 

To  implement  the  combination  of  two  real-time  sensor  video,  the 
following  functional  elements  are  needed  between  sensors  and  display  device 
(see  Figure  19):  the  preprocessor  (including  the  registration  computer);  the 
processor;  and  the  video  mixing/synchronization.  The  preprocessor  provides 
the  formatting  necessary  to  convert  the  sensor  data  to  common  computational 
and  display  parameters  and  also  determines  the  errors  in  image  translation. 
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rotation  and  magnification  for  registration  corrections.  The  processor 
function  consists  of  the  processing  operations  which  are  performed  on  the 
sensor  video  for  image  enhancing  or  cuing  purposes.  (Figure  19  shows  a 
shopping  list  of  processing  operations  which  could  implemented.  ) The  video 
mixing  and  synchronization  mixes  the  processed  video  from  the  radar,  FITR, 
and  TV  sensors  and  generates  a composite  video  output.  All  the  functions  are 
controlled  by  a microprocessor. 

A sizeable  number  of  possible  correction  functions,  processing 
operations,  and  mixing  techniques  are  included  in  the  general  system  of 
Figure  19  for  the  sake  of  completeness.  In  practice,  however,  an  operational 
system  will  undoubtedly  implement  only  a limited  number  corresponding  to 
the  desired  flexibility,  required  accuracy,  and  permissable  system  complexity 
of  the  intended  application. 

The  following  subsections  present  a more  detailed  description  of  the 
hardware  needed  for  function  implementation. 

6.  d.  I Preprocessor/Registration  Computer 

The  preprocessor  function  includes  the  analog  to  digital  conversion, 
automatic  boresighting  of  the  sensors,  the  radar  scan  conversion  to  TV  scan 
line  format,  and  the  warping  of  the  EO  imagery  from  an  azimuth /elevation 
format  to  a radar  az.imuth/range  format  or  the  inverse.  The  preprocessor 
also  includes  a programmable  symbol  generator  for  overlaying  symbols  on 
sensor  video. 

Digital  processing  is  preferred  over  analog  processing  because  of 
the  necessity  for  flexibility  and  control.  Another  advantage  of  digital 
processing  is  stability  and  repeatability  in  an  experimental  implementation, 
so  that  subsequent  evaluation  can  be  well  established.  The  digital  implemen- 
tation of  the  processor  functions  is  easily  interfaced  with  the  control  micro- 
processor to  provide  control  of  all  internal  variables,  i.  e.  , varying  gain, 
threshold,  and  processor  functions  implemented.  However,  it  should  be 
noted  that  rapid  advances  are  being  made  in  analog  circuit  technology 
(e.  g.  , charge  coupled  devices)  and  the  actual  mechanization  should  be 
selected  at  the  time  of  development. 

The  analog  to  digital  conversion  of  the  video  data  is  provided  for  the 
radar,  FLIR,  and  TV.  Since  only  two  imaging  sensors  are  displayed  at  any 
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time,  only  two  A/D  converters  are  required.  One  A/D  converter  is 
provided  internally  in  the  modular  digital  scan  converter  (MDSC)  which  will 
be  described  shortly.  A second  A/D  is  provided  and  is  multiplexed  with  the 
MDSC  A/D  to  provide  two  digitized  sensor  inputs.  The  multiplexing  functions 
are  provided  for  the  A/D  converter,  registration  computer  or  boresight 
correlator,  and  the  MDSC. 

The  boresight  correlator  (registration  computer)  corrects  azimuth 
and  elevation  boresight  errors  between  two  sensors,  one  sensor  acting  as  the 
reference  sensor.  Since  only  two  imaging  sensors  are  displayed  simultane- 
ously, only  one  boresight  correlator  is  required  with  multiplexed  inputs. 

Two  versions  of  boresight  correlator  have  been  developed  at  Hughes  Aircraft 
Company.  One  is  of  minimal  complexity  but  is  relatively  insensitive  to 
rotational  errors  between  sensors.  The  other  implementation,  which  is 
capable  of  computing  rotational  errors,  is  a factor  of  three  more  complex. 
Roth  boresight  correlators  operate  by  crosscorrelation  of  the  processed 
videos. 

The  boresight  offset  between  the  reference  sensor  and  the  second 
input  video  source  is  corrected  in  the  MDSC.  The  MDSC  provides  horizontal 
element  and  vertical  line  delays  to  adjust  the  offset  between  the  two  selected 
sensors.  Assuming  the  use  of  the  less  complex  correlator,  rotational  errors 
may  be  corrected  by  correlating  several  points  in  the  two  sensor  images  and 
then  calculating  the  rotational  error  in  the  microprocessor.  Video  rotation 
cannot  be  performed  in  real  time  in  the  MDSC  without  difficulty.  Therefore, 
the  rotational  error  should  be  applied  directly  to  the  sensor  orientation  to 
compensate  for  the  error  if  high  update  rate  sensors  are  involved.  If  stored 
imagery  and  low  update  rate  sensors  are  used,  then  the  correction  can  be 
applied  at  the  MDSC.  In  addition  to  determining  rotation  error,  correlation 
at  several  points  also  allows  magnification  or  FOV  errors  to  be  computed 
and  then  corrected  in  the  MDSC. 

The  radar  scan  conversion  converts  the  radar-type  raster  scans  at 
low  update  rates  to  a TV  format  at  TV  update  rates.  A version  of  the 
fv^DSC  developed  under  contract  to  the  Air  Force  Avionics  Laboratory,  is 
ideally  suited  to  this  application.  The  MDSC  is  controlled  by  a micro- 
processor and  can  provide  appropriate  flexibility  and  system  control.  The 
warping  of  an  EO  azimuth/elevation  format  to  the  radar  azimuth/range 
format  can  be  achieved  by  using  the  appropriate  scan  converter  memory 
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loading  algorithm.  Furthermore,  the  scan  converter  can  perform  the 
boresight  correlation  offset  correction  as  well  as  FOV  compensation  and  line 
rate  matching. 

If  the  two  sensors  operate  at  different  TV  line  rates  (say,  525  versus 
87  5 lines),  the  sensor  video  can  be  loaded  into  the  scan  converter  in  one  line 
standard  and  read  out  in  another.  Therefore,  the  one  sensor  video  can  be 
matched  in  line  rate  to  the  other.  By  passing  the  data  through  the  scan 
converter  memory,  the  horizontal  and  vertical  axis  can  be  scaled  to  adjust 
the  one  sensor  FOV  to  match  the  reference  sensor.  The  boresight  offset 
can  be  used  to  offset  the  center  position  of  the  video  in  the  scan  converter 
memory  to  compensate  for  the  boresight  error. 

Correction  for  intensity  nonlinearities  and  shading  can  be  provided 
digitally  via  programmable  read  only  memory  (PROM),  to  provide  a table 
lookup  mapping. 

The  microprocessor  bidirectional  serial  data  bus  can  be  used  to  pro- 
vide system  mode  control.  The  serial  bus  would  control  the  processing 
functions  employed. 

A symbol  generator  can  also  be  included  in  the  system.  The  symbol 
generator  is  programmable  and  controlled  by  the  microprocessor.  Based 
on  inputs  from  supplementary  sensors  such  as  warning  receivers,  MTI, 
laser  spot  tracker/designator/ ranger,  and  stored  or  linked  threat  data,  the 
microprocessor  instructs  the  symbol  generator  to  overlay  symbology  at 
appropriate  positions  on  the  sensor  video  display. 

Provision  can  also  be  made  for  displaying  the  output  of  automatic 
target  cuing  devices  which  are  presently  in  development.  These  sophisticated 
processors  are  being  designed  to  detect  and  discriminate  road,  structure, 
and  vehicle  targets.  When  such  a target  has  been  detected,  discriminated, 
and  located,  the  required  target  type  and  position  information  can  be  trans- 
ferred to  the  microprocessor.  The  microprocessor  would  then  provide 
the  symbol  generator  with  the  appropriate  instructions  to  provide  a symbolic 
overlay  of  the  target.  If  a tank  was  detected,  for  example,  the  symbol 
generator  would  display  a coded  symbol  at  the  coordinates  of  the  detected 
tank,  overlaid  perhaps  on  the  FLIR  sensor  imagery. 
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6.  Z.  Z Processor 


The  processor  functions  include:  I ) video  thresholding  for  placing 

sensor  number  1 highlights  on  sensor  number  Z video,  or  for  inserting 
sensor  number  1 video  in  saturated  or  dark  areas  of  sensor  number  Z imagery; 
Z)  scene  modulation  sampling  for  inserting  sensor  number  1 video  in  low 
detail  areas  of  sensor  number  Z;  3)  edge  enhancement  for  outlining  single 
sensor  video,  or  for  overlaying  outlines  of  sensor  number  1 imagery  on 
sensor  number  Z imagery;  4)  level  slicing  for  outlining  areas  of  constant 
brightness  on  a single  sensor  display,  or  for  overlaying  sensor  number  1 
outlines  on  sensor  number  Z imagery;  and  5)  color  selection  for  false 
coloring  of  each  gray  shade  in  a level-sliced  image,  or  for  color-coding  the 
sensor  information  in  each  of  the  above  4 combinations.  The  microprocessor 
selects  the  function  implementation  as  well  as  setup  of  control  variables. 
Implementation  of  each  technique  is  as  follows: 

Gain/Polarity 

Gain  control  is  provided  digitally  via  PROM  as  a table  look  up  multiply, 
as  shown  in  Figure  ZO.  The  polarity  is  simply  a multiplexing  between  the 
sensor  data  or  its  inverse. 

Threshold 

The  threshold  function,  as  shown  in  Figure  Zl,  selects  and  compares 
the  digitized  sensor  number  1 video  with  a threshold  determined  by  the 
microprocessor.  The  threshold  is  subtracted  from  the  sensor  video  to 
remove  all  video  below  (or  above)  the  threshold  from  the  sensor  video. 

For  highlighting  purposes,  the  video  number  1 exceeding  the  threshold 
is  outputted,  with  the  remaining  scanned  data  set  to  zero,  and  overlaid  or 
suitably  combined  with  sensor  number  Z video. 

For  fill-in  purposes,  the  microprocessor  sets  the  threshold  level 
such  that  video  number  1 above  a certain  "dark"  level  is  outputted  and 
video  number  Z "fills  in"  the  areas  below  the  thresholded  levels.  Or, 
video  number  L below  a certain  "white"  level  is  outputted  ar.d  video  number  2 
fills  in  the  areas  above  that  threshold  level. 
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Figure  20.  Polarity  and  gain  control. 
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Figure  21.  Threshold  for  highlighting  or  fill-in  of 
dark/bright  areas. 
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Scene  Modulation  Sampling 


In  this  technique,  the  sensor  video  is  sampled  to  determine  modulation. 
Low  modulation  areas  are  then  replaced  by  higher  modulation  video  from 
another  sensor.  The  low  modulation  areas  are  detected  by  accumulating 
the  variation  from  the  average  value  of  sensor  video  number  1.  When  a pre- 
determined threshold  is  not  exceeded,  the  sensor  video  is  replaced  by 
sensor  video  number  2 in  this  area. 

The  area  of  modulation  sampling  is  formed  as  a window  of  approxi- 
mately four  elements  by  four  lines.  The  window  slides  horizontally  and 
vertically  through  the  display  raster  until  the  entire  image  is  scanned.  The 
low  modulation  replacement  shown  in  Figure  2.2.  accumulates  the  video  from 
four  elements  of  the  present  and  past  TV  line.  The  accumulator  adds  the 
newest  video  element  and  subtracts  the  video  four  elements  delayed.  The 
two  accumulators  are  summed  to  form  the  mean  video  within  the  window. 
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Figure  22.  Scene  modulation  sampling  for  low 
modulation  replacement. 
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The  mean  is  subtracted  from  the  value  of  the  center  element  and  the  absolute 
value  is  accumulated  to  provide  a measure  of  the  modulation.  The  modula- 
tion factor  is  compared  to  a threshold  controlled  by  the  microprocessor, 
and  the  output  video  from  another  sensor  is  switched  in  when  the  threshold 
is  not  exceeded. 

Another  high  threshold  is  set  (by  the  microprocessor)  to  detect  targets 
in  low  modulation  areas.  If  the  video  minus  the  mean  exceeds  the  higher 
threshold,  the  original  video  element  is  inserted  into  the  second  sensor  video 
to  indicate  the  detected  target  position.  Thus,  a reasonably  failsafe  circuit 
is  provided  to  eliminate  the  loss  of  a peak  target  in  a low  modulation  area. 

Edge  Enhancement 

Edge  enhancement  is  achieved  by  a differential  operator  which  detects 
edges  in  the  video  scene.  Using  the  Roberts  operator  described  in  Sec- 
tion 5.  3.4,  edge- like  variations  occurring  horizontally,  vertically,  and 
diagonally  are  detected  and  the  video  gain  is  increased  proportionally  to  the 
edge  variation.  This  function  results  in  the  outlining  of  man-made  features 
which  are  contained  in  the  sensor  video. 

The  edge  enhancement  function  shown  in  Figure  23  forms  a two-element 
by  two-line  sliding  window  The  ROM  performs  a matrix  multiply  to  detect 
horizontal  and  vertical  boundaries  Where  boundaries  are  detected,  the  video 
gain  is  increased  to  enhance  the  edges.  The  microprocessor  selects  the 
amount  of  gain  (or  other  function)  applied  to  the  edge  regions. 

Level  Slicing 

Level  slicing  requires  that  multiple  thresholds  be  applied  to  the 
sensor  video.  To  obtain  false  color,  each  amplitude  slice  must  be  encoded 
at  a different  color.  As  shown  in  Figure  24,  the  level  slicing  function 
compares  the  sensor  video  with  multiple  thresholds  from  the  microprocessor. 
Each  threshold  forms  a window  which  can  be  used  to  encode  the  video  ampli- 
tude between  thresholds  into  a different  color.  The  threshold  value  is  sub- 
tracted from  the  video  to  remove  the  bias  from  the  output.  A ROM  multiply 
increases  the  gain  of  the  windowed  video  to  a full  scale  output.  The  output 
multiplexer  selects  between  the  thresholded  video,  normal  video  or  a 
no-output  condition. 
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False  Color  Select 


The  availability  of  color  display  allows  each  sensor  output  to  be 
artifically  encoded  into  a selected  color  output.  The  false  color  select 
function  shown  in  Figure  25  encodes  the  sensor  video  to  any  or  all  ideo 
color  outputs.  The  selection  is  under  microprocessor  control. 

6.  1.  3 Video  Combination 

Possible  combination  techniques  include  summing  the  two  video 
sources  pixel  by  pixel,  replacing  video  number  1 pixels  with  selected  pixels 
from  video  nunber  1,  interlacing  the  two  videos,  or  alternating  their 
presentation  at  a selectable  rate. 

In  the  alternation  scheme,  the  function  shown  in  Figure  26  switches 
from  one  sensor  to  another  by  multiplexing  between  the  two  sensors  at  an 
integer  number  of  field  times.  The  vertical  sync  is  counted  down  and  the 
microprocessor  selects  the  appropriate  frequency  from  the  set  of  60, 

30,  15  and  7.  5 Hertz. 


Figure  25.  False  color  select. 
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Figure  26.  Image  alternation/video  flashing. 

The  circuit  can  also  flash  or  flicker  target  video.  The  flashing 
function  compares  the  sensor  video  with  a threshold  determined  by  the  micro- 
processor. Sensor  video  exceeding  the  threshold  is  then  flickered  at  the 
same  selectable  rates  controlled  by  the  microprocessor. 

The  video  mixing  and  synchronization  function  mixes  the  two  processed 
sensor  videos  to  form  the  composite  video  waveform.  Compensation  is  pro- 
vided for  the  processing  delays  in  each  channel  depending  on  the  processing 
selected.  Standard  black  and  white  as  well  as  color  encoding  to  red-green- 
blue  standards  is  included.  The  appropriate  delay  is  also  provided  for  the 
video  blanking  and  synchronization  signals  for  composite  video  generation. 

6.  2.  4 System  Complexity 

Table  12  summarizes  the  various  elements  and  functions  included  in 
the  conceptual  simultaneous  sensor  presentation  system  as  well  as  element 
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TABLE  1 2.  SYSTEM  ELEMENT  COMPLEXITY,  SIZE, 
WEIGHT,  AND  POWER 


System  Elements 

Integrated 

Circuits 

Size 
Cu.  In. 

Power, 
W atts 

A/D  Converter 

50 

144 

0.  75 

10 

Modular  Digital  Scan  Converter 

1400 

1440 

21.0 

280 

• A/D 

• Scan  Converter 

• Symbol  Generator 

• Microprocessor 

Additional  Buffer  Memory 

1200 

1248 

18.  0 

240 

Boresight  Correlator-N  Line 

100 

192 

1.  5 

20 

Boresight  Correlator- Area 

400 

480 

6.  0 

80 

Positional  Dependent  Gamma  (2) 

40 

134 

0.  6 

8 

Gain  (2) 

10 

105 

0.  15 

2 

Polarity  (2) 

10 

105 

0.  15 

2 

Threshold  (2) 

20 

115 

0.  3 

4 

Modulation  Sample  (2) 

100 

192 

1.  5 

20 

Edge  Detect  (2 ) 

40 

134 

0.  6 

8 

False  Color  (2 ) 

20 

115 

0.  3 

4 

Pseudo  Color  (2) 

70 

163 

1.  0 

14 

Image  Alternation  (2) 

20 

115 

0.  3 

4 

Video  Mixing 

40 

134 

0.  6 

8 

System  (including  all 
enhancement  functions) 

■| 

• N-Line  Correlation 

3120 

2.5 

46.  8 

• Area  Correlation 

3320 

ICs 

2.  7 

Cu.  Ft. 

51.3 

Lbs. 

BB 

complexity,  expressed  in  terms  of  number  of  integrated  circuits  required. 
These  elements  correspond  to  approximately  47  of  the  modules  presently 
being  developed  for  the  Air  Force  Digital  Avionics  Information  System  (DAIS). 
Element  size,  weight  and  power  are  also  listed.  The  total  system  values 
represent  the  summation  of  all  elements  and  functions  listed.  In  practice, 
only  a limited  number  of  these  functions  would  be  implemented;  however, 
all  are  included  for  completeness.  Note  that  the  most  complex  elements 
are  the  microprocessor  controlled  digital  scan  converter  and  the  boresight 
correlator  for  determining  registration  errors.  The  simplification  of  the 
system  and  elimination  of  the  additional  buffer  memory  would  significantly 
decrease  system  complexity,  size,  weight  and  power. 

6.  3 DESCRIPTION  OF  TYPICAL  SYSTEM  OPERATION 

To  illustrate  the  order  of  operation  and  the  high  degree  of  flexibility 
of  the  system  described  in  6.  2 and  diagrammed  in  Figure  19,  two  typical 
sensor  combinations  are  examined  — FLIR/TV  for  use  at  short  ranges  and 
radar/FLIR  for  target  acquisition  at  stand-off  ranges. 

6.  3.  1 FLIR/TV  Simultaneous  Presentation 

In  this  example,  the  FLIR  sensor  is  assumed  to  be  the  primary  sensor, 
which  is  to  be  augmented  by  the  secondary  sensor,  i.  e.  , the  TV.  Thus,  the 
FLIR  input  represents  the  reference  or  baseline  and  the  TV  represents  the 
variable. 

Referring  back  to  the  system  diagram  of  Figure  19,  the  FLIR  video 
is  inputted  to  the  boresight  correlator  as  the  reference  video.  The  MDSC 
input  multiplexer  selects  the  TV  input  and  initially  performs  the  A/D  con- 
version. Since  the  TV  and  FLIR  are  compatible  in  scan  pattern  (azimuth/ 
elevation),  the  spatial  warping  algorithm  (first  order  approximation)  is  not 
implemented.  The  TV  data  is  loaded  into  the  scan  converter  memory  to  match 
the  FOV  and  scan  rate  of  the  FLIR.  The  converted  TV  video  is  inputted  to 
the  boresight  correlator.  The  correlator  measures  the  boresight  offset 
between  the  TV  and  FLIR  (horizontal  and  vertical),  the  magnification  differ- 
ence, and  the  rotational  error.  The  horizontal  and  vertical  offsets  and 
magnification  differences  are  sent  to  the  MDSC  where  the  corrections  are 
implemented  digitally  via  position  offsets  in  the  digital  scan  converter 
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memory.  Because  rotational  error  correction  is  very  complex  for  high 
data  rate  orthogonal  scan  systems,  it  is  recommended  that  rotational  errors, 
if  significant,  be  corrected  in  the  sensor  mechanical  or  readout  alignment. 
The  result  of  the  preprocessing  function  then  is  to  provide  two  sensor  inputs  . 
which  are  boresighted,  and  have  common  FOV  and  scan  rates. 

Moving  now  to  the  processing  function,  the  FLIR  video  is  inputted  as 
Sensor  Number  1 and  the  TV  video  is  inputted  as  Sensor  Number  2.  The  gain 
and  polarity  required  for  the  desired  processing  algorithm  is  selected  for  the 
FLIR,  and  a similar  selection  is  made  for  the  TV.  As  an  example  from  the 
shopping  list  of  processing  functions  shown  in  Figure  19,  the  scene  modula- 
tion sampling  operation  (Figure  24)  can  be  used  for  the  FLIR  such  that  low 
detail  areas  are  replaced  by  the  TV  video.  The  edge  detection  operator 
(Figure  23)  can  also  be  applied  to  the  FLIR  to  enhance  target  edges.  The 
false  color  select  can  encode  the  enhanced  FLIR  edges  or  outlines  in  red, 
while  the  baseline  FLIR  imagery  is  displayed  in  black  and  white.  The  low 
modulation  areas  which  have  been  replaced  by  the  TV  can  be  encoded  in 
blue/green. 

The  processed  video  outputs  are  summed  in  the  video  mixing/ 
synchronization  after  the  processing  delays  are  compensated.  If  symbology 
is  desired  to  represent  data  from  supplementary  sources  such  as  warning 
receivers,  GMTI,  laser  spot  tracker,  stored  and  linked  threat  data,  or 
automatic  target  cuing  devices,  the  symbol  generator  output  would  also 
be  mixed  to  form  the  composite  video  output. 

The  operations  described  above  apply  to  a multisensor  target 
acquisition  pod  containing  both  a FLIR  and  TV,  or  to  the  synergistic  use  of 
acquisition  pod  sensors  and  missile  seeker  sensors.  Many  attack  aircraft 
do  not  presently  carry  both  FLIR  and  TV  due  to  space,  weight  and  cost 
limitations.  The  B-52  (EVS)  and  B-57  (Tropic  Moon  III)  are  equipped  with 
both  FLIR  and  LLLTV.  The  A-6E  and  A-7E  carry  FLIR  only,  and  the  F- 16, 
F-18,  and  B- 1 may  carry  FLIR  only.  A possible  means  of  obtaining  TV 
imagery  in  an  F-18  with  FLIR-only  target  acquisition  would  be  the  utilization 
of  a TV  seeker  in  one  of  its  missiles  (if  the  seeker  has  or  were  given  some 
slewing  capability). 

A similar  combination  of  acquisition  pod  sensor  and  missile  seeker 
sensor  might  be  envisioned  for  the  Pave  Knife /Imaging  1R  Maverick  or  the 
TRAM/TV  Maverick. 
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6.  3.2  Radar/FLIR  Simultaneous  Presentation 

In  this  example,  a doppler  beam- sharpened  (DBS)  patch  radar  mode  is 
the  primary  or  reference  sensor  and  a forward  looking  infrared  (FLIR)  imaging 
system  is  the  secondary  or  augmenting  sensor.  Because  of  gross  incompati- 
bilities between  these  two  sensors,  the  analysis  and  design  presented  here 
represent  one  of  the  most  difficult  problems  of  simultaneous  presentation. 

One  of  the  aims  of  this  analysis  is  to  show  that  a careful  consideration  of  the 
specific  nature  of  the  sensors  leads  to  a simple  r block  diagram  with  resultant 
savings  in  system  hardware  and  complexity  at  no  loss  of  performance. 

The  DBS  radar  image  covers  a radial  segment  with  approximately 
12°  FOV  and  approximate  ground  dimensions  of  4 mi.  by  4 mi.  at  a distance 
of  20  nmi.  as  shown  in  Figure  27.  The  proc  essing  time  for  each  full  frame 
is  on  the  order  of  4 sec.  The  FLIR  images  cover  a smaller  FOV,  ranging 
from  3°  to  5°  down  to  1.5°,  and  each  FLIR  frame  is  received  in  approxi- 
mately 1/30  sec.  Thus,  the  FLIR  data  enter  at  a much  faster  rate  than  the 
radar  but  correspond  to  a smaller  FOV.  The  FLIR  sensor  must  then  be 
stepped  or  scanned  in  an  orderly  fashion  to  cover  the  same  footprint  as  the 
DBS  patch  and  the  resulting  FLIR  images  overlaid  on  top  of  the  DBS  radar 
imagery,  as  shown  in  Figure  27.  In  this  figure,  a 3°  FOV  is  assumed  for 
FLIR  vs.  12°  for  DBS  radar,  such  that  approximately  12  FLIR  image  patches 
are  needed  to  cover  the  radar  footprint.  The  major  problem  here  is  to 
determine  the  correct  position  of  eac  h FLIR  patch  as  it  arrives  in  relation 
to  the  DBS  radar  footprint  as  well  as  in  relation  to  the  adjacent  FLIR  patch. 

In  addition,  there  may  be  an  unintentional  relative  motion  between  each  FLIR 
patch  and  the  DBS  radar  image,  so  that  image  correlation  is  necessary  for 
accurate  frame  registration.  Other  sources  of  error  which  need  to  be 
compensated  include  static  (systematic)  sensor  misalignment. 

A system  diagram  of  a processor  to  combine  these  images  is  shown 
in  Figure  28.  The  radar  data  undergo  a scan  conversion  process  with  a 
coordinate  transformation  (polar  to  rectangular  mapping)  and  storage  in  a 
scan  conversion  buffer  memory  before  display.  The  buffer  memory  also 
serves  as  a refresh  so  that  a flicker-free  30  frames-per- second  image  can 
be  seen  on  a low  persistence  display.  Further  image  processing  for  enhance- 
ment follows  the  buffer. 
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Figure  <28.  Radar /FLIR  processor  block  diagram. 

In  addressing  the  problem  of  overlaying  the  sensor  images,  it  is 
assumed  that  gross  magnification  and  rotation  problems  are  removed  by  prior 
processing  or  as  part  of  the  correlator  as  described  in  Sections  5.  1 and  6.  <2. 
An  efficient  system  requires  an  additional  image  buffer  memory,  since  the 
FLIR  data  frames  enter  sequentially  as  noted  in  Figure  <27  and  each  frame 
is  synchronized  with  the  DBS  radar  to  scan  areas  in  an  overlapping  patch  work 
work  manner  as  shown.  Each  FLIR  patch  is  adjusted  to  the  correct  coordinate 
system  and  stored  in  the  buffer  memory.  The  first  FLIR  patch  is  correlated 
against  the  DBS  radar  in  correlator  1 of  Figure  28  and  is  then  positioned 
correctly  in  the  buffer  for  overlap.  When  additional  FLIR  patches  arrive 
(every  l/30th  second)  they  can  be  positioned  in  one  of  two  ways.  The  first 
way  is  to  use  correlator  1 (DBS  radar  vs.  FLIR)  and  to  insert  the  patch  in  its 
appropriate  location  on  the  display.  The  correlation  search  time  should  be 
small  for  this  problem  because  the  FLIR  is  generally  synchronized  to  a 
particular  local  area  of  the  DBS  radar  and  only  a small  correction  is  deter- 
mined by  the  correlator.  A second  method  is  to  correlate  the  edges  of  over- 
lapping FLIR  patches  in  a smaller  correlator  (2  on  the  diagram  in  Figure  28) 
and  to  insert  them  in  the  appropriate  position.  In  either  method,  any  over- 
laps between  FLIR  patches  are  resolved  by  always  inserting  new  data 
(destroying  any  old  image)  into  the  FLIR  buffer  memory. 
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Many  FLIR  frames  can  be  processed  in  the  time  for  one  DBS  radar 
frame.  For  the  example  here,  4 sec/radar  frame  — 1/30  sec/FLIR  frame 
= 120  FLIR  frames  entering  in  the  time  for  one  DBS  frame.  In  Figure  28, 

there  are  12  FLIR  patches,  so  each  patch  can  be  renewed  with  new 
data  as  many  as  10  times  per  DBS  radar  frame.  This  first  method  (using 
correlator  1 and  discarding  correlator  2)  has  an  advantage  because  suc- 
ceeding FLIR  frames  are  not  tied  to  the  origin  established  by  FLIR  patch  1. 
Thus,  strong  relative  motion,  misalignment  and  other  errors  can  be 
removed  as  best  as  possible  before  the  FLIR  data  is  inserted.  Careful 
synchronism  of  the  FLIR  patch  scanner  with  the  DBS  radar  is  required  to 
minimize  correlation  search  time.  In  practice,  the  buffer  memories  in 
Figure  28  may  be  part  of  the  scan  conversion  system  (including  coordinate 
transformation)  or  of  the  correlator,  but  they  are  drawn  separately  for 
clarity.  The  decision  to  combine  the  buffer  refresh  memory  with  various 
blocks  depends  on  existing  hardware  subsystems  such  as  the  modular  digial 
scan  converter  that  may  be  available,  as  well  as  other  digital  design 
considerations. 

For  almost  all  practical  systems  it  is  necessary  to  have  two 
separate  coordinate  transformation  and  buffer  memory  systems  to  null 
out  errors,  misalignment  and  motion  effects  as  noted,  unless  the  radar 
is  already  provided  in  a TV  format  (as  in  the  F-  18  radar).  Without  the 
additional  coordinate  transform  and  buffer  memory  system,  it  is  necessary 
to  identify  the  number  of  FLIR  patch  locations  from  Figure  27  very  accurately 
a priori  in  relation  to  the  DBS  image  for  accurate  image  overlay. 

An  alternate  implementation,  less  accurate  but  far  simpler,  is  to 
eliminate  the  second  buffer  memory  for  FLIR  data  and  instead  use  a symbol 
generator  to  indicate  hot  spots.  The  FLIR  video  for  each  patch  or  scan 
location  is  thresholded,  and  the  location  of  the  FLIR  hot  spots  are  trans- 
ferred to  the  control  microprocessor  to  generate  symbols  by  means  of  the 
symbol  generator.  The  symbols  are  mixed  with  the  radar  video  at  the 
hot  spot  locations  and  in  this  manner  target  points  are  identified  without 
need  of  an  additional  scan  converter  memory. 


The  basic  requirements  are  that  the  FLIR  and  DBS  radar  be 
correlated  at  some  initial  position,  and  the  FLIR  can  then  be  stepped  along 
in  an  overlapping  patch  manner.  The  FLIR  gimbal  angles  and  hot  spot  (x,  y) 
coordinates  at  each  FLIR  line-of- sight  position  are  the  only  data  that  need  to 
be  accumulated  during  this  process  for  transfer  to  the  microprocessor.  This 
technique  is  feasible  because  the  FOV  reading  rates  for  the  FLIR  exceed  the 
DBS  reading  rates  by  a factor  of  over  100  such  that  the  FLIR  mosaic  of 
patches  can  consist  of  an  array  of  as  many  as  10  x 10  sample  FOV's.  The 
F LIR  hot  spots  can  appear  as  color- coded  (red)  dots  on  a black/white  or 
green  radar  display  or  as  highest  intensity,  flashing  dots  on  a monochrome 
display. 

The  operations  described  above  apply  to  the  synergistic  use  of  an 
aircraft  mounted  multimode  radar  with  a pod  mounted  FLIR  target  acqui- 
sition system.  Such  combinations  are  appropriate  for  intermediate  to 
stand-off  range  target  acquisition  and  weapon  delivery  concepts  as  discussed 
in  Section  4.  0 

6.4  CONCLUSIONS 

A general  system  implementation  has  been  presented  for  the  simul- 
taneous display  of  a variety  of  data.  The  system  accepts  inputs  from 

1.  Real-time  sensors,  such  as  radar,  FLIR,  TV,  laser  spot 
trackers /rangers,  warning  receivers,  GMTI,  etc.; 

<2.  A priori  data  stored  in  the  aircraft  data  base,  such  as  reconnais- 
sance imagery  from  radars,  IR  sensors,  aerial  camera,  etc.  , 
cartography,  and  prebriefed  waypoint/target/threat  coordinates; 

3.  Data-linked  target  imagery  and  updated  target/threat  coordinates. 

The  general  system  is  designed  to  handle  the  most  complex  case, 
i.  e.  , the  combination  of  two  real-time  sensors  such  as  radar  and  FLIR  or 
FLIR  and  TV.  It  can  thus  readily  handle  the  simpler  cases  such  as  radar 
plus  stored  imagery  and  radar  plus  cartography.  The  systemallows  for  a 
multiplicity  of  processing  functions  including  coordinate  transformation, 
scan  conversion,  correction  for  geometric  distortions  and  intensity  non- 
linearities,  determination  and  correction  of  registration  errors  in  x,  y 
translation,  magnification,  and  rotation,  controllable  gain  and  polarity,  a 
variety  of  selec  table  image  processing  and  enhancement  operations, 
selectable  color  encoding,  and  a number  of  video  mixing  techniques. 


Implementation  methods  have  been  presented  for  each  of  these 
functions  for  completeness  and  to  demonstrate  the  extensive  flexibility  and 
capability  which  could  be  designed  into  a multiscnsor  display  system.  In 
practice,  however,  an  operational  system  will  undoubtedly  implement  only 
a limited  number  of  these  operations  in  an  intended  application,  depending  on 
the  types  of  information  to  be  combined,  general  performance  and  accuracy 
requirements,  desired  flexibility,  and  permissable  hardware  size,  weight 
and  power.  For  example,  fine  corrections  may  not  be  required  for 
geometric  distortions  or  brightness  nonlinearities.  Strict  tolerances  may 
not  be  required  for  image  registration  (in  fact,  misregistration  of  images 
may  actually  be  advantageous).  One  or  two  simple  image  enhancement 
operations  may  be  sufficient  to  extract  the  salient  information  required 
from  one  sensor  for  display  on  a second  sensor  image.  Thus,  an  austere 
system  with  a high  performance  level  may  be  achieved  by  incorporating 
only  the  minimal  essential  elements  with  accuracies  and  tolerances  ade- 
quate for  the  specific  application. 

A comparison  of  the  relative  complexity  of  mechanizing  various 
simultaneous  presentations  is  given  in  Table  13.  The  primary  sensor  is 
shown  for  each  phase  of  the  interdiction  mission,  as  well  as  a number  of 
secondary  sensors  or  data  to  augment  the  primary  sensor.  For  each 
combination  of  primary  sensory  and  secondary  sensor  or  data,  the  key 
system  elements  required  to  implement  that  combination  are  listed.  The 
lowest  common  multiple  of  these  elements  is  sufficient  to  characterize  a 
system  capable  of  being  utilized  in  every  mission  phase.  Also  listed  are  a 
quantitative  estimate  of  relative  system  complexity,  an  estimate  of 
"availability"  or  time  to  a flyable  brassboard,  and  those  combinations  which 
appear  to  warrant  concentrated  effort  in  the  near  future. 

It  should  be  noted  that  much  of  the  hardware  shown  in  the  general 
system  implementation  may  already  be  onboard  a high  performance  tactical 
aircraft.  For  example,  a digital  scan  converter  may  be  associated  with  or 
contained  within  the  radar  equipment  (as  in  the  F-18).  A control  micro- 
processor, a central  data  bus,  and  a symbol  generator  may  also  be  part  of 
the  avionics.  All  of  the  above  are  key  elements,  for  example,  in  the  Navy's 
Advanced  Integrated  Display  System  (AIDS)  and  the  Air  Force's  Digital 
Avionics  Information  System  (DAIS).  , 
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TABLE  13.  COMPARISON  OF  CANDIDATE  MULTISENSOR 
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A boresight  correlator  may  also  be  an  onboard  requirement  to 


1.  Coordinate  the  radar  and  target  acquisition  FLIR  (or  TV) 
for  navigation  updates  and  fire  control  functions,  and 

2.  To  boresight  the  missile  sensor  (imaging  IR,  quasi- imaging 
IR,  TV,  or  imaging  microwave)  to  the  pilot's  target  acquisition 
sensor  (FLIR  or  TV)  for  target  hand-off. 

Therefore,  there  is  a well-defined  and  widely  acknowledged  need  for  weapon 
system  requipment  to  correlate  very  diverse  types  of  imagery  in  real-time. 
Because  the  simultaneous  presentation  of  several  sensor  data  on  a single 
display  also  requires  the  correlation  of  dissimilar  imagery,  the  same 
hardware  and  software  being  developed  for  target  hand-off  is  directly 
applicable  to  that  needed  for  implementation  of  the  multisensor  display 
concept. 

Thus,  the  addition  of  a simultaneous  presentation  capability  need 
not  mean  a major  addition  of  avionics.  Also,  all  of  the  elements  identified 
in  the  general  system  represent  equipment  that  is  already  available  or 
presently  in  development. 

In  estimating  the  complexity  of  this  equipment,  emphasis  was 
placed  on  Hughes'  related  engineering  experience.  A sizable  number  of 
the  elements  in  the  general  system  are  presently  in  development  at  Hughes, 
including 

• The  An/UYK  30  microprocessor  developed  with  company 
funds  and  Air  Force  and  Navy  support, 

• The  microprocessor- controlled  modular  digital  scan  converter 
(including  symbol  generator  and  central  data  bus)  developed 
for  AFAL  and  flight- tested  in  1976, 

• The  in-raster  symbol  generator  developed  for  NADC  and 
delivered  in  197  6, 

• The  N-line  boresight  correlator  and  the  more  sophisticated 
area  boresight  correlator  developed  on  company  support. 

Hughes  is  also  the  prime  contractor  on  the  DAIS  program  to  build 
an  integrated  cockpit  display  system  for  the  Air  Force's  "hot  bench"  facility. 
Thus,  the  suggested  implementations  for  the  multisensor  display  system 
are  based  on  state-of-the-art  equipment  whose  capabilities  and  interfaces 
are  well  understood. 


7.  0 OBSERVATIONS  AND  CONCLUSIONS 


This  study  has  examined  the  simultaneous  presentation  of  multisensor 
data  on  a single  display  in  terms  of  potential  utility  to  the  pilot  and  estimated 
complexity  of  implementation. 

The  following  paragraphs  summarize  pertinent  observations  which 
were  made  during  the  conduct  of  this  study,  major  conclusions  which  emerged 
as  a result  of  the  study,  and  essential  efforts  which  are  required  in  the  future 
to  confirm  and  quantify  those  conclusions. 

1.  From  the  interdiction  mission  analysis,  it  is  apparent  that  the 
pilot  in  a single-seat,  high-performance  attack  aircraft  is  faced 
w'ith  an  increasing,  if  not  overwhelming,  amount  of  information 
which  must  he  perceived,  assimilated,  and  mentally  integrated 
for  required  judgments  to  be  made  and  appropriate  action  to  be 
taken.  These  tasks  are  intensified  during  critical  portions  of 
the  mission  such  as  target  acquisition  and  weapon  delivery,  and 
are  further  intensified  during  night  operations. 

2.  From  the  sensor  capabilities /tar get  characteristics  survey, 
the  advantages  of  multiple  sensors  in  target  acquisition  and 
weapon  delivery  are  evident.  This  is  due  to  the  multi-attribute 
nature  of  targets  which  makes  them  detectable  in  a number  of 
different  spectral  regions  (passively  or  actively)  and  thus  by  a 
number  of  different  sensors.  The  attributes  can  be  time  critical 
so  that  if  one  signature  ceases,  another  signature  may  remain 
detectable  by  means  of  multisensor  capability. 

3.  From  the  analysis  of  operator  tasks  as  a function  of  mission 
phase  and  the  subsequent  determination  of  the  baseline  sensors 
which  could  be  utilized  to  perform  those  tasks,  the  following 
observations  can  be  made. 

• There  is  no  one  sensor  which  provides  sufficient  information 
to  act  as  the  principal  or  lead  sensor  throughout  the  mission. 

• Instead,  a single  primary  sensor  typically  emerges  for  each 
mission  phase  to  provide  the  basic  information  required  at 
that  time.  A set  of  secondary  sensors  and  collateral  data 
are  available  to  supplement  the  primary  sensor  information 
and  thereby  provide  target  cuing,  target  verification,  threat 
warning  and  other  valuable  data  to  assist  in  target  acquisition 
or  defensive  action. 

• Also,  there  is  typically  no  smooth  transition  from  the  pri- 
mary sensor  information  of  one  phase  to  the  primary  sensor 
information  of  the  next  phase.  Conflicts  and  gaps  can  and  do 
exist  between  sensors. 
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The  simultaneous  presentation  of  multisensor  data  on  a single 
display  represents  one  possible  technique  for 

a.  Assisting  the  operator  in  integrating  a multiplicity  of 
essential  data. 

b.  Increasing  the  probability  of  successful  target  acquisition 
and  weapon  delivery,  and 

c.  Improving  aircraft  survivability. 

A major  premise  in  the  simultaneous  sensor  presentation  con- 
cept is  that  primary  sensors  and  secondary  sensors /collateral 
data  can  be  synergistically  combined  within  useful  time  periods, 
i.  e.  , mission  phases.  However,  the  differences  between  sensor 
capabilities  and  the  gaps  encountered  when  transitioning  from 
one  primary  sensor  (mission  phase)  to  the  next  must  somehow 
be  resolved,  either  by  introducing  a priori  knowledge,  by  utilizing 
additional  sensors,  or  by  further  refining  target  identity  and 
location  by  some  other  means.  This  is  based  on  the  following 
observ  at  ions : 

• A priori  data,  i.  e.  , target  data  and  enroute  threat  data 
compiled  before  take-off  for  prebriefed  missions  and  entered 
into  the  A/C  data  base,  can  be  of  major  assistance  within 
each  mission  phase  and  greatly  contribute  to  pilot  perfor- 
mance and  overall  mission  success. 

• The  primary  real-time  sensor  for  each  mission  phase  can 
typically  be  aided  by  one  or  more  additional  sensors  which 
are  able  to  provide  some  type  of  augmenting  information  at 
the  primary  sensor's  operational  range.  Each  of  the  sensors 
combined  would  thus  contribute  its  strongest  range-compatible 
target  acquisition  trait. 

• Advanced  target  location  and  identification  techniques  are 
being  pursued  which  may  facilitate  the  target  acquisition 
problem  in  the  future.  These  include  a)  the  processing 
of  two  sensor  outputs  such  as  synthetic  aperture  radar 
returns  and  ELINT  system  line-of- bear ing  for  greatly 
improved  accuracy  and  timeliness  of  emitter  targeting, 
weapons  allocation  or  threat  avoidance  maneuvers;  and  b)  the 
processing  of  sensor  video  for  feature  extraction  and  pattern 
recognition  of  rivers,  roads  and  eventually,  tanks,  trucks, 
etc.  , for  target  cuing  purposes. 

During  the  conduct  of  the  study,  several  promising  simultaneous 
sensor /sensor  and  sensor  /a  priori  data  display  candidates 
emerg  1 for  use  in  an  interdiction  mission.  The  sources  con- 
sidered or  simultaneous  presentation  included 

a.  Real-time  sensors  such  as  radar,  FLLR,  TV,  GMTI,  radar/ 
laser  warning  receivers,  laser  spot  trackers /rangers,  etc. 
and 

b.  A priori  (stored)  data  such  as  target  and  known  threat 
coordinates,  cartography,  and  reconnaissance  imagery  of 
the  target  area  (photo,  radar,  IR,  laser  linescan,  etc.  ). 


Marty  of  these  combinations  were  simulated  by  means  of  static 
imagery  and  qualitatively  evaluated  in  terms  of  potential  utility 
to  the  operator.  The  most  promising  combinations  which  emerged 
are  listed  in  Table  14  as  a function  of  mission  phase.  The  table 
associates  the  primary  imaging  sensor  for  each  phase,  the  secon- 
dary imaging  sensor,  and  a number  of  supplementary  sensors  and 
collateral  data  which  can  be  displayed  by  means  of  overlaying 
symbology  on  primary  sensor  video.  Provision  was  also  made 
for  future  incorporation  of  automatic  feature  extraction  and  target 
cuing  devices,  whose  output  can  also  be  displayed  as  a symbol 
over 1 ay. 

Those  imagery  candidates  which  appear  to  warrant  further 
investigation  include:  the  simultaneous  presentation  of  radar  and 
cartography  for  navigation;  radar  and  stored  target  imagery  (or 
maps)  for  preliminary  acquisition  of  large,  fixed  targets;  radar 
and  FL1R  hot  spots  for  preliminary  acquisition  of  targets  of 
opportunity;  and  FLIR  and  TV  for  closer  range  target  recognition 
and  identification  (and  correlation  hand-off  of  targets). 

6.  Implementation  of  the  most  promising  simultaneous  sensor 

presentations  can  be  effected  using  hardware /software  elements 
w'hich  either  have  been  or  presently  are  being  developed.  All 
operations  should  be  capable  of  being  performed  in  real  time  with 
sufficient  accuracy  for  this  application.  Definite  advantage  can 
be  taken  of  avionic  subsystems  and  processors  already  onboard 
such  as  the  A/C  mission  computer,  the  radar  scan  converter,  the 
boresight  correlator  used  for  target  handoff  to  missile  sensors, 
etc. 

During  the  course  of  the  study,  a general  implementation  concept 
v'as  identified  for  taking  two  video  inputs  (selected  from  real-time 
radar,  FLIR,  and  TV  sensors,  or  stored /linked  imagery),  making 
them  format  and  scan-line  compatible,  correcting  for  registra- 
tion errors,  performing  any  desired  enhancement  operations, 
and  mixing  the  two  registered,  processed  video  in  an  optimum 
manner  for  simultaneous  display.  This  general  concept  is 
modular  in  nature  and  highly  programmable.  It  performs  the 
various  correction  operations  sequentially  and  then  provides  for 
a number  of  selectable  image  processing  and  display  techniques. 

hot  all  elements  in  the  general  implementation  are  required  for 
any  single  combination.  Thus,  the  system  could  be  greatly 
simplified  depending  on  the  particular  multisensor  combination 
desired.  Those  elements  required  to  implement  each  of  the 
recommended  simultaneous  presentations  of  Table  14  were 
identified  during  the  study  and  rated  in  terms  of  complexity  and 
estimated  date  of  flyable  brassboard  model  availability.  The 
most  complex  elements  are  the  microprocessor  - controlled 
digital  scan  converter  and  the  image  registration  computer 
(boresight  correlator),  both  of  which  are  presently  in  develop- 
ment at  Hughes. 
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7.  While  the  most  promising  simultaneous  presentations  can  be 
implemented  using  a monochrome  display,  color  coding  of  the 
various  sensor/data  inputs  may  assist  the  operator  in  per- 
forming certain  tasks.  On  a monochrome  display,  secondary 
sensor  highlights  or  a priori  data  may  lie  distinguished  from 
primary  sensor  video  by  being  displayed  at  highest  intensity 
and  perhaps  intermittently  flashed;  also,  shape-coded  symbology 
can  be  used.  If  a color  display  were  provided,  sensor  highlight 
information,  graphics,  and  symbology  could  be  color-coded  when 
overlaid  on  primary  sensor  imagery.  A human  factors  evalua- 
tion would  be  required  to  quantify  any  improvement  in  operator 
performance  using  multicolor  presentations  versus  monochrome. 

Present  cockpit  displays  are  monochrome;  however,  there  are 
ongoing  efforts  in  the  area  of  militarized  multicolor  displays 
such  as  ruggedizing  the  shadow-mask  CRT  and  improving  the 
performance  of  the  penetration  phosphor  CRT.  The  development 
of  the  latter  would  provide  a Z to  3 color  display  with  gray  scale 
capability,  which  would  be  sufficient  for  this  application,  and 
most  likely  easier  to  achieve  than  a militarized  full  color,  full 
gray  level  display.  Future  efforts  may  take  advantage  of 
advanced  flat  panel  display  technology  such  as  the  matrix- 
addressed  liquid  crystal  pictorial  display. 

8.  In  conclusion,  a qualitative  evaluation  of  an  extensive  number  of 
potential  simultaneous  data  presentations,  based  on  estimated 
utility  to  the  operator  and  complexity  of  implementation,  has 
resulted  in  several  candidate  combinations  which  are  recom- 
mended for  further  study.  In  terms  of  usefulness  to  the  opera- 
tor, these  candidates  offer  the  potential  of  improved  target 
acquisition,  weapon  delivery,  and  survivability  without  an 
increase  in  task  load.  In  terms  of  implementation,  the  most 
complex  hardware  and  processing  elements  required  are 
already  scheduled  for  inclusion  in  advanced  attack  aircraft 
avionic  s. 

9.  In  regard  to  recommended  future  efforts,  a significant  problem 
was  encountered  in  this  study  and  in  previous  efforts  uncovered 
in  the  literature  survey,  i.  e.  , the  lack  of  usable,  coincident 
coverage,  multispectral  imagery  (static  or  dynamic).  Without 
such  imagery,  no  quantitative  evaluation  of  performance  improve- 
ment associated  with  multisensor  displays  can  be  made.  There- 
fore, three  major  recommendations  for  future  effort  emerge: 

• That  a data  base  of  simultaneous,  coincident  coverage 
multisensor  imagery  be  acquired  using  real-time,  high 
performance  sensors,  including  radar,  FITR  and  TV. 

• That  a brassboard  model  of  key  elements  in  the  multisensor 
display  concept  be  assembled,  using  off-the-shelf  com- 
ponents and  incorporating  the  various  image  enhancement 
functions,  to  simulate  the  most  promising  multisensor 
combinations  recommended  in  Table  14  and  to  determine  the 
optimum  interfacing  of  the  subsystems. 
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• That  a human  factors  evaluation  of  the  recommended 

multisensor  presentations  be  conducted  (initially  static, 
ultimately  dynamic)  to  1 ) quantify  the  anticipated  improve- 
ment in  operator  performance  over  single  or  sequential 
sensor  use;  quantify  the  permissable  tolerances  in  cor- 
rections for  geometric  distortions,  intensity  nonlinearities, 
and  misregistration  (slight  misregistration  of  images  may 
actually  enhance  target  acquisition);  3)  evaluate  the  effects 
of  the  various  image  enhancement  operations  on  target 
acquisition;  and  4)  investigate  the  anticipated  performance 
improvement  in  false  color  coding  of  the  multisensor  inputs. 

10.  Should  these  three  recommendations  lie  implemented,  it  is 

expected  that  the  simultaneous  presentation  system  capabilities 
will  be  sufficiently  well- characterized  that  the  concept  can  be 
readily  extended  and  applied  to  a variety  of  missions.  It  is 
also  expected  that  system  operation  for  these  missions  will 
be  highly  automated.  On  a prebriefed  mission,  for  example, 
the  sequencing  of  simultaneous  presentations  can  be  preselected, 
typically  as  a function  of  mission  phase.  A'so,  the  processing 
and  image  enhancing  operations  associated  with  each  sequential 
simultaneous  presentation  can  be  preprogrammed.  Thus, 
operator  interaction  and  task  load  can  be  minimized,  while 
allowing  for  operator  override  in  all  cases. 

The  system  of  the  future  can  be  envisioned  as  totally  adaptive 
to  the  mission  objectives,  mission  profile,  and  operator 
requirements.  All  simultaneous  presentation  functions  will 
oe  coordinated  by  the  mission  computer.  The  selected  types 
of  targets,  the  anticipated  backgrounds  in  which  those  targets 
are  embedded  or  surrounded,  the  selected  mission  profile,  and 
meteorological  conditions  encountered  along  that  profile  will  be 
entered  into  the  mission  computer.  Stored  algorithms  will 
determine  the  optimum  combination  of  primary /secondary 
sensors  for  each  mission  phase,  will  automatically  sequence  and 
switch  the  combinations  as  a function  of  phase  throughout  the 
mission,  and  will  select  the  optimum  processing  operations  for 
image  enhancement  and  image  understanding  as  a function  of 
sensor  combination  and  mission  phase.  Operator  interactive 
control  of  the  system  will  be  provided,  as  well  as  override 
capability. 

This  automated,  adaptive  simultaneous  presentation  system  of 
the  future  can  free  the  operator  to  concentrate  on  the  essential 
tasks  of  mission  management,  c ritic  al  decision  making,  and 
offen sive /defensive  systems  supervision.  The  study  has 
determined  that  such  a system  of  the  future  is  technically 
feas  ible . 
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8.0  GLOSSARY  OF  TERMS  AND  ABBREVIATIONS 


AAA 

A- A 

ADC 

A FAC 

A - C i 

AGR 

AIM 

CAP 

CAS 

CBU 

CP 

CPU 

DBS 

ECM 

ELS 

EO 

FAC 

FAE 

FUR 

FOV 

FTI 

FTT 

GMTI 

GMTT 

GPS 

HARM 

HMD 

HUD 

IFF 

ILS 

INS 


Anti-Aircraft  Artillery 

Air-to- Air 

Air  Data  Computer 

Airborne  Forward  Air  Controller 

Air-to-Ground 

A ir- to- Ground  Ranging 

Air  Interceptor  Missile 

Combat  Air  Patrol 

Close  Air  Support 

Cluster  Bomb  Unit 

Checkpoint 

Central  Processor  Unit 
Doppler  Beam  Sharpening 
Electronic  Counter  Measures 
Emitter  Location  System 
Electro- Optical 
Forward  Air  Controller 
Fuel-Air  Explosive 
Forward-Looking  Infrar.  d 
Field  of  View 

Fixed  Target  Indication  Radar 

Fixed  Target  Track 

Ground  Moving  Target  Indication 

Ground  Moving  Target  Track 

Global  Positioning  System 

High  Velocity  Anti- Radiation  Missiie 

Helmet- Mounted  Display 

Head-Up  Display 

Identification,  Friend  or  Foe 

Instrument  Landing  System 

Inertial  Navigation  System 


1 37 


I 


L 


IP 

IK 

JTIDS 

LATV 

LLLTV 

LOC 

LOS 

LOR  AN 

LWK 

MTI 

NOTH 

OTH 

PE  LSS 

PPI 

PRF 

PTS 

RAM 

RBGM 

RWR 

RWS 

SAM 

SAR 

SLAR 

STT 

TA 

TACAN 

TF 

TOA/DMF 

TRAM 

TWS 

TEREC 

TASES 

UGS 

USS 

VT  AS 

WRCS 


Identification  Point 
Infrared 

Joint  Tactical  Information  Distribution  System 

Laser- Aided  Television 

Low  Light  Level  Television 

Line  of  Communication 

Line  of  Sight 

Long  Range  Navigation  (System) 

Laser  Warning  Receiver 
Moving  Target  Indication  Radar 
Non-Cooperative  Target  Recognition 
Over  The  Horizon 

Precision  Emitter  Location  and  Strike  System 

Plan  Position  Indicator 

Pulse  Repetition  Frequency 

Photogrammetric  Targeting  System 

Random  Access  Memory 

Real  Beam  Ground  Map 

Radar  Warning  Receiver 

Range  While  Search 

Soviet  Air  Missile 

Synthetic  Aperture  Radar 

Side- Looking  Airborne  Radar 

Single  Target  Track 

Terrain  Avoidance 

Tactical  Aircraft  Navigation  (System) 

Terrain  Following 

Time  of  Ar ri v il / Distan ce  Measurement  Equipment 
Target  Recognition/ Attack  Multisensor 
Track  While  Scan 

Tactical  Electronic  Reconnaissance  Set 

Tactical  Airborne  Signal  Exploitation  System 

Unattended  Ground  Sensor 

Unattended  Sea  Sensor 

Visual  Target  Acquisition  System 

Weapon  Release  Computer  System 


1 38 


I 


I 


APPENDIX  A 

SURVEY  OF  MULTISENSOR  STUDY  LITERATURE 

A key  objective  in  the  investigation  of  simultaneous  sensor 
presentations  is  improved  operator  task  performance.  Any  simultaneous 
sensor  display  concept  must  therefore  provide  sufficient  potential  for  opera- 
tor performance  improvement  if  it  is  to  be  considered  for  development,  test- 
ing and,  eventually,  implementation  in  operational  military  aircraft. 

A number  of  simultaneous  sensor  presentation  techniques  are  identi- 
fied in  Section  4 of  this  report.  It  is  not  possible  to  analytically  determine 
what  performance  improvement  might  be  achieved  with  the  various  techniques 
identified.  Empirical  quantitative  evaluation  is  the  necessary  method  to 
assess  the  performance  improvement  that  can  be  achieved  with  multisensor 
display  techniques.  Such  an  evaluation  was  beyond  the  scope  of  this  study. 

In  this  study,  two  techniques  were  employed  to  provide  a preliminary  assess- 
ment of  the  potential  of  the  simultaneous  presentation  concept  for  performance 
improvement.  These  two  techniques  were  1)  a review  of  the  literature  and 
Z)  a qualitative  evaluation  of  selected  simultaneous  sensor  presentation 
techniques  by  means  of  synthesized  static  imagery.  The  literature  review 
of  behavioral  (operator  performance)  studies  in  the  area  of  simultaneous 
sensor  presentation  is  presented  below.  Samples  of  the  synthesized  imagery 
are  presented  in  Section  4. 

Fourteen  opeiator  performance  studies  relevant  to  multisensor  dis- 
play were  culled  from  Defense  Documentation  Center  and  Hughes  bibliographic 
literature  searches.  The  presentation  of  these  studies  is  organized  into  two 
major  categories;  1)  multiple  real-time  sensors,  and  2)  one  real-time  sensor 
and  one  or  more  non -real -time  sensors.  Non -real -time  sensors  are  defined 
here  as  pre-mission  information,  such  as  target  location  data,  maps,  photos, 
and  other  reconnaissance /intell  igence  data.  Presentation  technique  was 
used  as  a second  level  categorization  of  the  studies  reviewed.  Simultaneous 
multiple  sensor  presentation  on  a single  display,  simultaneous  multiple 
sensor  presentation  on  multiple  displays,  and  sequential  multiple  sensor 
presentation  on  a single  display  were  the  three  presentation  technique 
categories  used. 
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Radar  was  the  primary  sensor  in  a large  percentage  of  the  studies 
reviewed.  This  is  due  to  a fair  amount  of  research  interest  in  briefing  and 
reference  materials  and  target  pre -designation  (non -real -time  sensors)  for 
air-to-ground  radar  strike  systems. 

A.  I MULTIPLE  REAL-TIME  SENSORS 

A.  1.  1 Simultaneous  Presentation  on  a Single  Display 

Two  studies  were  located  which  i nvesti gated  simultaneous  sensors 
presented  on  a single  display.  The  more  interesting  of  the  two  studies 
(Stinnett,  Leonard,  and  Farehert,  1969)  investigated  five  display  presenta- 
tion modes  for  common  aperture  TV  and  IR  video.  The  second  study  (Green 
and  Kalensher,  197Z)  investigated  0.4880  to  10.6  micrometer  multispectral 
line  scan  imagery  for  tactical  target  cueing. 

The  Stinnett,  Leonard,  and  Farehert  (1969)  study  evaluated  simul- 
taneous TV  and  FLIR  sensor  presentation  techniques  for  vehicle  detection 
and  recognition.  The  two  sensors  had  a common  aperture  and  a 5 -degree 
field  of  view  with  100  by  100  element  resolution.  The  five  display  presenta- 
tion modes  studied  were: 

1.  TV  only  in  black  and  white 

Z.  IR  only  in  black  and  white 

3.  TV  and  IR  superimposed  in  black  and  white 

4.  TV  displayed  in  blue-green,  IR  inserted  by  contour  fill  displayed 
in  red 

5.  Same  as  4)  with  IR  alternated  (on-off)  at  a Z Hz  rate. 

A truck,  a Ford,  and  a Volkswagon,  each  located  in  three  background 
scenes,  provided  the  sensor  imagery  for  the  study.  The  vehicle  targets 
were  driven  across  a dirt  road  at  10  mps  in  the  three  scenes;  these  moving 
targets  were  recorded  on  video  tape  at  ranges  between  1900  and  3300  feet 
for  use  in  the  study.  Thirty  observers  participated  in  the  study  — six 
observers  for  each  of  the  five  display  presentation  modes. 

The  results,  time  and  probability  of  target  detection  and  recognition, 
revealed  that  operator  target  detection  performance  was  not  differentially 
affected  by  the  five  dPnlay  modes.  In  other  words,  the  observers  detected 
the  targets  about  equally  well  regardless  of  whether  TV,  IR,  or  combined 
TV  and  IR  were  displayed. 
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Target  recognition  performance  (time  and  probability)  was,  however, 
differentially  affected  by  the  five  display  modes  evaluated.  Figure  A-l 
shows  target  recognition  time  and  probability  performance  with  the  five 
display  modes,  IR  only  was  superior  to  TV  only,  and  TV  and  IR  super- 
imposed was  inferior  to  IR  only.  The  best  target  recognition  performance 
was  achieved  with  the  two  modes  in  which  TV  was  displayed  in  blue-green 
and  IR  was  inserted  in  red.  Performance  was  slightly  better  when  the  2 Hz 
alternation  was  used.  Comparison  between  IR  only  and  TV  with  IR  inserted 
shows  a 57  percent  increase  in  target  recognition  probability  and  a 17  per- 
cent decrease  in  time  to  recognize  using  the  multisensor  TV  and  IR  mode. 

The  results  of  the  Stinnett,  Leonard,  and  Farehert  (1969)  study 
clearly  indicate  the  potential  for  improved  operator  performance  with 
simultaneous  sensors.  It  is  also  clear  that  to  achieve  a performance 
improvement,  simultaneous  sensors  must  be  properly  formatted  and  dis- 
played — improperly  superimposed  TV  and  IR  can  and  does  degrade  perform- 
ance compared  to  single  sensor  presentation. 

It  should  be  pointed  out  that  there  are  limitations  in  the  study.  The 
use  of  three  background  scenes  and  three  targets  must  have  made  it  unreal- 
istically easy  for  the  observers  to  search  the  scenes  and  detect  and  recog- 
nize the  targets.  The  use  of  six  observers  per  condition  is  too  small  a 
sample  to  assume  equivalent  groups.  There  is,  therefore,  some  uncertainty 
whether  observed  differences  among  display  modes  were  due  to  observers 
within  the  five  groups  of  six  display  modes  or  the  display  modes  themselves. 
A final  criticism  is  the  relatively  small  sensor  field  of  view  and  short  range, 
probably  necessary  because  of  the  low  sensor  resolution,  which  resulted  in 
scenes  with  an  unrealistically  small  terrain  coverage,  particularly  for  a 
search  and  detection  task.  Notwithstanding  these  limitations,  the  study 
provides  the  only  operator  performance  information  available  on  simultan- 
eous TV  and  IR  video  presented  on  a single  display. 

The  Green  and  Gale  (1972)  study  was  a rooftop  laboratory  demonstra- 
tion of  a multi  spectral  scanner.  A laser  line  scanner  produced  imagery  at 
six  different  wavelengths  from  0.4880  to  10.6  micrometers,  and  an  IR  line 
scanner  produced  imagery  in  the  spectral  region  from  1 to  5.  5 and  8 to 
14  micrometers.  A specially  designed  processor  was  programmed  to  "find1' 
military  objects  Icanvas  and  army  uniforms)  in  a target  scene  that  contained 
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DISPLAY  MODES  LEGEND 


MODE  1 

TV  ONLY 

MODE  2 

IR  ONLY 

MODE  3 

TV  AND  IR  SUPERIMPOSED 

MODE  4 

TV  IN  BLUE-GREEN,  IR 
INSERTED  IN  RED 

MODE  5 

SAME  AS  MODE  4 WITH 
2 Hz  ALTERNATION  RATE 

DISPLAY  MODES 


Figure  A-l.  Target  recognition  results  from  Stinnett,  Leonard, 
and  Farehert  (1969)  study. 


foliage,  sand,  ponds,  a dirt  road,  and  a number  of  military  equipment  items. 
The  target  scene  was  constructed  in  a lot  adjacent  to  the  rooftop  laboratory. 

The  imagery  was  recorded  on  black  and  white  as  well  as  color  film. 
False  color  images  were  generated  by  assigning  different  sensor  wavelength 
bands  to  different  colors.  For  example,  the  cue  canvas  materials,  spectral 
signatures  for  olive  drab  canvas  were  assigned  to  red;  all  other  spectral 
signatures  were  presented  in  green. 

According  to  the  authors,  "The  results  of  the  work  were  very  encour- 
aging. " Of  the  15  canvas  and  army  uniform  targets,  the  processor  correctly 
found  12  of  the  targets.  There  were  four  false  detections.  The  preliminary 
results  obtained  indicate  that  a multispectral  sensor  system  can  be  developed 
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that  can  automatically  find  selected  targets  in  real-time.  The  targets  found 
by  the  automatic  system  could  be  cued  to  an  operator  for  more  detailed 
examination  or  other  action.  The  basic  concept  appears  to  be  feasible  and 
the  potential  for  improved  target  detection  by  a man -machine  system  looks 
pr  omis  ing. 

A.  1.  2 Simultaneous  Presentation  on  Multiple  Displays 

Studies  by  Snyder,  Earl,  Wyman,  and  Sturm  (1966)  and  Leachtenauer , 
Jones,  Iden,  and  Cook  (1968)  investigated  simultaneous  presentation  of 
multiple  sensors  on  multiple  displays.  Snyder,  Earl,  Wyman,  and  Sturm 
investigated  1 1 combinations  of  radar,  IR,  TV,  and  direct  vision.  The 
radar  had  500-feet  range  resolution  and  3.6-degree  azimuth  resolution. 

The  IR  had  a 45-degree  field  of  view  with  3 -mill iradian  resolution,  the  TV 
sensor  was  a standard  525-line  system  with  a 27 -degree  field  of  view.  The 
color  direct  vision  imagery  was  shown  at  a 53-degree  field  of  view.  The 
four  sensors  were  optically  displayed  via  16 -mm  motion  picture  film  on 
separate  dedicated  displays  in  a single -seat  cockpit  simulator.  (The  direct 
visual  scene  was  back -projected  onto  a screen  outside  the  cockpit.  The  TV, 
IR , and  radar  sensor  imagery  w as  back -projected  onto  screens  in  the 
cockpit.  ) 

The  44  Air  Force  fighter  pilots  w'ho  were  operators  in  the  study  were 
tasked  to  recognize  prebriefed  location  known  targets,  e.  g.  , bridges, 
reservoirs,  and  industrial  complexes.  Briefing  packets  were  used  for  pre- 
trial study  and  reference  during  the  trials. 

Eigure  A -2  shows  the  percent  correct  target  recognition  achieved 
with  the  11  combinations  of  forward  view  (direct  vision),  TV,  IR,  and  radar 
sensors  investigated.  Best  performance  was  achieved  with  forward  view/IR 
and  forward  view/radar  combinations  (65  to  70  percent  recognition);  radar 
alone  produced  the  poorest  performance  (9  percent  correct  recognition). 
Although  the  forward  view/IR  and  forward  view/radar  sensor  combinations 
resulted  in  larger  percentage  of  targets  recognized  than  TV  only  or  forward 
view  only,  the  differences  (15  to  20  percent)  were  not  statistically  reliable. 
There  is,  therefore,  some  uncertainty  about  the  performance  advantage  of 
simultaneous  multiple  sensors  displayed  separately. 
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SENSOR  MODE  COMBINATIONS 


NOTE  ANY  TWO  SENSOR  MODES 

NOT  UNDER  THE  SAME 
LINE  ARE  SIGNIFICANTLY 
DIFFERENT  P < 0 05 

LEGEND 

F V = FORWARD  VIEW 
IR  = INFRARED 
R = RADAR 
TV  = TELEVISION 


Figure  A-2.  Target  recognition  results  from  Snyder,  Earl,  Wyman, 

and  Sturm  (1966)  study. 

According  to  the  ai'thors,  "The  results  clearly  indicate  that  the 
addition  of  IR  or  radar  or  both  to  any  sensor  combination  which  includes 
either  visual  sensing  or  TV  does  not  significantly  affect  performance  either 
by  TV  or  by  direct  vision.  Apparently  the  pilot  devotes  primary  attention 
to  the  visual  or  TV  scene  if  it  is  available  and  only  supplements  either  of 
these  displays  slightly  by  reference  to  the  IR  or  radar.  If  ne'ther  the  TV 
nor  the  direct  vision  mode  is  available,  IR  is  more  useful  than  radar  — at 
least  for  the  targets  used  in  this  study.  " 

Leachtenauer , Jones,  Iden,  and  Cook  (1968)  investigated  the  property 
of  one  type  of  sensor  to  enhance  critical  objects  and  focus  attention  on 
corresponding  areas  of  imagery  obtained  from  a different  sensor  for  appli- 
cation to  intelligence  extraction.  Four  sensor  combinations  were 
investigated; 

1.  Fixed  target  side -looking  radar  (SLR)  to  focus  on  photography 

2.  SLR  moving  target  indication  (MTI)  to  focus  on  photography 
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3.  SLR  to  focus  on  IR 

4.  IR  to  focus  on  photography. 

Three  presentation  orders  of  the  two  sensors  (focuser  sensor  and  focused 
upon  sensor)  were  also  investigated  in  the  study.  These  were  1 ) focuser 
first,  focuser  second,  and  focuser  and  focused  upon  sensor  images  simul- 
taneous. The  actual  sensor  imagery  was  APQ-10Z  SLR,  AN/AAS-18A  IR, 
and  KS-7Z  forward  oblique  aerial  photography. 

Nine  image  interpreter  operators  viewed  the  imagery  combinations 
on  a light  table.  Time,  accuracy,  completeness,  and  efficiency  of  operator 
image  interpretation  task  performance  were  measured. 

Except  for  one  condition  (fixed  target  SLR  to  focus  on  photography), 
focusing  techniques  did  not  improve  operator  performance,  and  the  authors 
state  that  the  positive  SLR/photo  results  may  have  been  an  experimental 
artifact.  Interpretation  time  was  shorter  when  the  focuser  was  presented 
first;  however,  the  shortest  times  occurred  when  no  focuser  was  used. 

Although  the  two  studies  which  investigated  simultaneous  sensor 
presentation  on  multiple  displays  represent  a small  sample  and  are  limited 
in  application  by  the  sensor  imagery  available  in  the  mid-  to  late  1960's, 
the  indication  is  clear.  Presenting  multiple  simultaneous  sensors  on 
multiple  displays  is  not  likely  to  appreciably  increase  the  probability  of  the 
operator  finding  targets  and  will  increase  the  time  to  find  targets.  At  best, 
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imaging  radar  to  cue  an  electro -optical  (EO)  sensor.  This  concept  takes 
advantage  of  the  long  range  sensing  inherent  to  the  radar  and  the  high  resolu- 
tion characteristic  of  EO  sensors. 

Humes,  Craig,  Poplawski,  and  Hershberger  (1974)  conducted  a 
laboratory  study  of  synthetic  aperture  radar  (SAR)  image  quality  require- 
ments for  cueing  an  EO  sensor.  The  operator's  task  in  the  study  was  to 
locate  a predefined  target  area  on  the  radar  ground  map.  The  accuracy,  in 
ground  feet,  of  locating  and  designating  the  target  area  was  used  to 
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analytically  evaluate  the  feasibility  of  using  SAR  for  cueing  EO  sensors. 
Analyses  were  performed  to  relate  'operator  radar  des ignationaccuracy  to 
EO  sensor  field  of  view,  slant  range  to  target,  and  transverse  distance 
across  the  EO  sensor  field  of  view  as  illustrated  in  Figure  A -3. 

The  conclusions  drawn  from  the  laboratory  research  and  analysis 
are  as  follows: 

1.  To  gain  any  advantage  in  a combination  cueing  radar /EO  system, 
the  radar  cueing  designation  error  should  not  exceed  1500  feet. 
This  can  be  achieved  with  20-  to  40-foot  resolution  radars  using 
area  designation.  Direct  designation  of  a target  or  designation 
of  an  offset  cue  point  yields  the  smallest  error. 

2.  Small  cueing  errors  will  allow  small  EO  fields  of  view  which  can 
increase  the  probability  of  recognizing  EO  targets  at  long  range. 

3.  Electro -optical  fields  of  view  on  the  order  of  3 degrees  or  less 
are  required  for  long  range  ( 30,000  feet)  recognition  of  small 
tactical  targets.  (Meteorological  effects  at  such  ranges  were  not 
considered  in  the  analysis.  ) 

Sequential  presentation  of  real-time  sensors  may  well  be  an 
important  multi-sensor  application.  Radar  as  an  EO  cueing  sensor  appears 
to  be  one  of  the  more  promising  techniques.  Man- in-the-loop  simulation 
should  be  performed  to  determine  the  extent  of  improved  operator  tactical 
target  acquisition  per  formance  w ith  sequential  sensor  presentation. 


TRANSVERSE  DISTANT*  ACROSi  EG  SENSOR  FIEID  OF  VIEW,  FEET 


Figure  A-3.  Relationship  among,  radar  designation  accuracy,  EO  sensor 
field  of  view,  slant  range  to  ground,  and  transverse  distance  across 

EO  sensor  field  of  view. 
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a.  2 one;  re;ai.-time:  sensor  and  one  or  more  non-real-time 

SENSORS 

A.  2.  1 Simultaneous  Presentation  on  a Single  Display 

Three  papers  that  treated  the  use  of  non -real -time  sensor  information 
with  a real-time  sensor  to  improve  operator  performance  were  found  during 
the  literature  search.  Two  of  the  papers  dealt  with  superimposing  cartog- 
raphy on  ground  map  radar  video.  The  third  paper  investigated  the  effect 
of  target  predesignation  on  operator  target  recognition  performance. 

Habbarth  < 1957,  1958)  compared  side-by-side,  sequential,  and  super- 
imposition presentation  of  1:500,000  scale  Sectional  Aeronautical  Charts  with 
simulated  30-nautical  mile  range  scale  B-52  ground  map  radar  imagery. 

The  operators'  tasks  in  the  laboratory  studies  were  to  locate  and  designate 
prebriefed  radar  targets. 

Super  imposition  of  the  charts  over  the  radar  imagery  produced  the 
best  overall  time  and  accuracy.  Chart  presentation  mode  was  found  to 
interact  with  the  type  of  radar  target.  The  side-by-side  chart  and  radar 
presentation  technique  was  best  tor  highly  articulated  targets;  superimposi- 
tion  of  chart  and  radar  data  was  best  for  no  shoe*,  targets. 

f'erranti  Ltd.  has  developed  a combined  (superimposed)  radar/map 
display  for  the  Harrier  Aircraft  (Braid,  1971).  Matching  the  map  to  the 
radar  is  done  by  the  operator  who  slews  the  map  to  match  terrain  features 
on  the  radar.  The  matching  process  takes  a maximum  of  10  seconds. 

Airborne  evaluation  of  the  combined  radar/map  display  produced 
positive  results.  The  simultaneous  display  of  radar  and  map  data  eliminated 
gross  navigation  errors  and  permitted  positive  identification  of  radar 
features.  These  advantages  are  achieved  by  map  fill-in  of  the  radar  where 
shadowing  and  weak  returns  occur.  Another  benefit  is  that  one  display 
satisfies  both  radar  and  map  display  requirements. 

The  use  of  navigation  system  or  other  information  sources  to  provide 
target  predesignation  was  investigated  by  Sturm,  Snyder,  and  Wyman  (1966). 
The  effects  of  target  predesignation  on  both  direct  visual  and  radar  target 
search  were  investigated.  Predesignation  was  done  by  placing  a crosshair  at 
the  expected  location  of  the  target.  The  operators'  task  was  to  recognize 
prebriefed  large  fixed  targets.  Probability  of  correct  target  recognition 
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increased  20  percent  and  1 1 percent  for  the  visual  and  radar  targets, 
respectively,  when  target  prede signatior  was  used. 

For  the  three  papers  reviewed,  simultaneous  presentation  of  real- 
and  non- real-time  sensor  data  on  a single  display  resulted  in  improved 
operator  performance.  For  the  prebriefed  target  mission,  this  simultaneous 
sensor  application  appears  very  promising.  With  today's  sensor,  processing, 
and  display  technology,  it  would  seem  that  considerable  improvement  in 
operator  performance  could  be  achieved  by  integrating  real-  and  non-real- 
time sensor  data  on  a single  display. 

A.  2.  2 Simultaneous  Presentation  on  Multiple  Displays 

Several  laboratory  research  investigations  have  been  performed  to 
determine  the  benefits  of  and  the  design  requirements  for  briefing  and 
reference  material  used  with  high  resolution  ground  mapping  radar  (Cahill 
and  Luce,  1967;  Carel  and  Hershberger,  1 967;  Humes,  Craig,  Poplawski, 
Guerin,  and  Hershberger,  1974;  Kause,  1968;  McKechnie,  1968;  McKechnie, 
969;  Welch  and  McKechnie,  1964.  All  but  one  of  the  studies  (Welch  and 

l 

McKechnie,  1964)  showed  improved  performance  when  the  operators  were 
provided  briefing  and  reference  material  (maps,  charts,  reconnaissance 
photography,  prior  radar  coverage)  and/or  target  location  information. 
Generally  speak  ;.ng,  the  greater  the  similarity  between  the  non-real-time 
sensor  data  and  the  real-time  sensor  and  the  greater  the  information  about 
th*  target's  location,  the  greater  the  improvement  in  operator  target 
acquisition  performance. 

fn  the  Welc  h and  McKechnie  (1964)  study,  for  example,  the  operators 
were  required  to  identify  all  airfields,  bridges,  tank  farms,  power  lines, 
and  railroad  yards  on  moving  (3.  3 inches  per  minute)  side-looking  radar 
imagery.  Sectional  Aeronautical  Charts  were  the  briefing  and  reference 
materials  used.  For  the  charts  to  be  of  any  benefit,  the  operators  had  to 
find  targets  on  the  chart  provided  and  then  correlate  the  chart  to  the  radar 
image,  during  which  time  the  radar  image  was  moving  across  the  display. 

Such  a process  is  difficult  and  time  consuming.  It  is  not  surprising  that  the 
operators  did  no  better  at  identifying  targets  when  charts  were  available. 

Providing  specific  target  location  information  (target  cueing)  on  the 
chart  can,  however,  significantly  improve  the  operator's  performance.  In  a 
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study  procedurally  very  similar  to  Welch  and  McKechnie  (1964),  McKechnie 
(1^68)  found  that  circling  the  targets  on  charts  increased  the  probability  of 
correctly  finding  the  targets  by  39  percent. 

Simultaneous  presentation  of  real-time  and  non-real-time  sensor  data 
on  multiple  displays  can  be  beneficial  to  operator/ system  performance. 
Information  that  can  be  provided  from  non- real- ti me  sensors  possesses  the 
potential  for  improved  operator  real-time  sensor  task  performance.  Dis- 
playing such  non- real -time  sensor  data  on  a separate  reference  display  is 
. a satisfactory  approach.  However,  the  displayed  information  must  be  pre- 

sented in  such  a manner  that  the  operator  can  efficiently  utilize  it. 

A.  2.3  Sequential  Pr e sentation  of  Real-  and  Non-Real-Time  Sensor  Data 

Sequential  presentation  of  real-  and  non- real- time  sensor  data  does 
not  seem  to  be  a particularly  viable  multiple  sensor  presentation  technique. 
Non-real-time  sensor  data  may  be  presented  prior  to  the  real-time  sensor, 
but  it  should  be  retained  for  use  during  the  time  the  real-time  sensor  is 
being  displayed.  Straight  sequential  presentation  of  non- re al- time  and  real- 
■ time  sensor  information  is  likely  to  be  of  little  practical  utility.  This 

opinion  is  confirmed  by  a laboratory  research  study  (Hubbarth,  1957)  in 
which  sequential,  side-by-side,  and  superimposition  presentation  of  Sectional 
Aeronautical  Charts  and  simulated  B-52  ground  map  radar  were  evaluated. 
Sequential  presentation  resulted  in  the  poorest  operator  target  acquisition 
performance  of  the  three  techniques.  Straight  sequential  presentation  of 
non-real-time  and  real-time  sensors  can  therefore  be  eliminated  as  a major 
contending  technique  for  multisensor  presentation. 

A.  3 CONCLUSIONS  AND  RECOMMENDATIONS 

The  preceding  review  of  the  literature  indicates  that  multiple  simul- 
taneous sensor  presentation  techniques  do  hold  promise  for  improved  operator 
task  performance.  This  improved  performance  may  be  achieved  with  multi- 
ple real-time  sensors  as  well  as  combinations  of  real-  and  non-real-time 
sensors.  Simultaneous  multiple  sensors  presented  on  a single  display 
appears  to  be  the  best  presentation  technique,  however,  it  is  also  the  most 
challenging.  Simple  addition  of  two  imaging  sensors  will  most  likely  degrade 
operator  performance,  not  enhance  it.  Similarly,  displaying  real-time 
sensor  video  from  two  sensors  on  two  displays  is  not  likely  to  improve 

I 


149 


operator  task  performance.  The  key  is  integration  of  sensor  information, 
not  simple  increases  in  quantity  of  information.  Providing  more  information 
may  serve  only  to  increase  the  operator's  processing  time  and  make  his  task 
more  difficult. 

The  14  papers  reviewed  are,  in  sum,  di  scour  aging  — not  so  much  in 
individual  quality,  hut  rather  the  lack  of  definite  direction  in  developing  any 
particular  area.  The  14  papers  represent  a large  breadth  of  research  but 
little  depth.  Thus,  one  cannot  really  recommend  any  particular  application 
of  a simultaneous  sensor  presentation  technique.  Simultaneous  sensor 
presentation  techniques  are,  we  think,  a fruitful  area  for  improving  operator 
task  performance.  Better,  more  definitive  research,  however,  will  be 
required  before  such  techniques  are  adequately  defined  and  developed  for 
airborne  military  aircraft. 
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